Virtualized server systems and methods including scaling of file system virtual machines

ABSTRACT

Examples described herein include virtualized environments including a virtualized file server. Examples of file server virtual machine managers are described which may setup and/or configure file server virtual machines in accordance with a target number of connections and/or a target throughput for the virtualized file server. The virtualized file server may be scaled up, scaled down, scaled in, and/or scaled out.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of pending U.S. patent applicationSer. No. 15/833,391 titled “Virtualized Server Systems And MethodsIncluding Scaling of File System Virtual Machines”, filed Dec. 6, 2017.The aforementioned non-provisional application is hereby incorporated byreference in its entirety, for any purpose.

This application claims the benefit under 35 U.S.C. 119 of the earlierfiling date of U.S. Provisional Application No. 62/430,882 titled “LOADESTIMATION FOR VIRTUALIZED SERVERS”, filed Dec. 6, 2016. Theaforementioned provisional application is hereby incorporated byreference in its entirety, for any purpose.

This application claims the benefit under 35 U.S.C. 119 of the earlierfiling date of U.S. Provisional Application No. 62/430,877 titled“HYPERVISOR-AGNOSTIC VM DEPLOYMENT”, filed Dec. 6, 2016. Theaforementioned provisional application is hereby incorporated byreference in its entirety, for any purpose.

This application claims the benefit under 35 U.S.C. 119 of the earlierfiling date of U.S. Provisional Application No. 62/431,768 titled“VIRTUALIZED FILE SERVER”, filed Dec. 8, 2016. The aforementionedprovisional application is hereby incorporated by reference in itsentirety, for any purpose.

TECHNICAL FIELD

Examples described herein generally relate to file servers invirtualized environments.

BACKGROUND

A “virtual machine” or a “VM” generally refers to a specificsoftware-based implementation of a machine in a virtualizationenvironment, in which the hardware resources of a real computer (e.g.,CPU, memory, etc.) are virtualized or transformed into the underlyingsupport for the fully functional virtual machine that can run its ownoperating system and applications on the underlying physical resourcesjust like a real computer.

Virtualization may work by inserting a thin layer of software directlyon the computer hardware or on a host operating system. This layer ofsoftware contains a virtual machine monitor or “hypervisor” thatallocates hardware resources dynamically and transparently. Multipleoperating systems run concurrently on a single physical computer andshare hardware resources with each other. By encapsulating an entiremachine, including CPU, memory, operating system, and network devices, avirtual machine is completely compatible with most standard operatingsystems, applications, and device drivers. Most modern implementationsallow several operating systems and applications to safely run at thesame time on a single computer, with each having access to the resourcesit needs when it needs them.

Virtualization may allow one to run multiple virtual machines on asingle physical machine, with each virtual machine sharing the resourcesof that one physical computer across multiple environments. Differentvirtual machines can run different operating systems and multipleapplications on the same physical computer.

One reason for the broad adoption of virtualization in modern businessand computing environments is because of the resource utilizationadvantages provided by virtual machines. Without virtualization, if aphysical machine is limited to a single dedicated operating system, thenduring periods of inactivity by the dedicated operating system thephysical machine is not utilized to perform useful work. This iswasteful and inefficient if there are users on other physical machineswhich are currently waiting for computing resources. To address thisproblem, virtualization allows multiple VMs to share the underlyingphysical resources so that during periods of inactivity by one VM, otherVMs can take advantage of the resource availability to processworkloads. This can produce great efficiencies for the utilization ofphysical devices, and can result in reduced redundancies and betterresource cost management.

Furthermore, there are now products that can aggregate multiple physicalmachines, running virtualization environments to not only utilize theprocessing power of the physical devices to aggregate the storage of theindividual physical devices to create a logical storage pool wherein thedata may be distributed across the physical devices but appears to thevirtual machines to be part of the system that the virtual machine ishosted on. Such systems operate under the covers by using metadata,which may be distributed and replicated any number of times across thesystem, to locate the indicated data. These systems are commonlyreferred to as clustered systems, wherein the resources of the group arepooled to provide logically combined, but physically separate systems.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a clustered virtualization environment according tosome particular embodiments.

FIG. 1B illustrates data flow within an example clustered virtualizationenvironment according to particular embodiments.

FIG. 2A illustrates a clustered virtualization environment implementinga virtualized file server (VFS) 202 according to particular embodiments.

FIG. 2B illustrates data flow within a clustered virtualizationenvironment.

FIG. 3A illustrates an example hierarchical structure 300 of a VFSinstance in a cluster according to particular embodiments.

FIG. 3B illustrates two example host machines 200 a and 200 b, eachproviding file storage services for portions of two VFS instances FS1and FS2 according to particular embodiments.

FIG. 3C illustrates example interactions between a client 330 and hostmachines 200 a and 200 c on which different portions of a VFS instanceare stored according to particular embodiments.

FIG. 3D illustrates an example virtualized file server having a failovercapability according to particular embodiments.

FIG. 3E illustrates an example virtualized file server that hasrecovered from a failure of Controller/Service VM CVM-1 110 a byswitching to an alternate Controller/Service VM CVM-3 110 c according toparticular embodiments.

FIG. 3F illustrates an example virtualized file server that hasrecovered from failure of a FSVM by electing a new leader FSVM accordingto particular embodiments.

FIGS. 3G and 3H illustrate example virtualized file servers that haverecovered from failure of a host machine 200 a by switching to anotherController/Service VM and another FSVM according to particularembodiments.

FIGS. 4A and 4B illustrate an example hierarchical namespace 400 of afile server according to particular embodiments.

FIG. 4C illustrates distribution of stored data amongst host machines ina virtualized file server according to particular embodiments.

FIG. 5 illustrates an example method for accessing data in a virtualizedfile server according to particular embodiments.

FIG. 6 illustrates an example of how a file server ‘FS1’ may be deployedacross multiple clusters according to particular embodiments.

FIG. 7 is a block diagram of an illustrative computing system 700suitable for implementing particular embodiments.

FIG. 8 is a schematic illustration of a system including a controller VMhaving a file system virtual machine manager (FSVM manager) arranged inaccordance with embodiments described herein.

FIG. 9 is an illustration of a view of a user interface arranged inaccordance with examples described herein.

FIG. 10 is an illustration of a view of a user interface arranged inaccordance with examples described herein.

DETAILED DESCRIPTION

Particular embodiments provide an architecture for implementingvirtualized file servers in a virtualization environment. In particularembodiments, a virtualized file server may include a set of File ServerVirtual Machines (VMs) that execute on host machines and process storageaccess operations requested by user VMs executing on the host machines.The file server VMs may communicate with storage controllers provided byController/Service VMs executing on the host machines to store andretrieve storage items, such as files and folders, on storage devicesassociated with the host machines. The storage items may be distributedamongst multiple host machines. The file server VMs may maintain astorage map, such as a sharding map, that maps names or identifiers ofstorage items, such as folders, files, or portions thereof, to theirlocations. When a user application executing in a user VM on one of thehost machines initiates a storage access operation, such as reading orwriting data from or to a storage item or modifying metadata associatedwith the storage item, the user VM may send the storage access operationin a request to one of the file server VMs on one of the host machines.In particular embodiments, a file server VM executing on a host machinethat receives a storage access request may use the storage map todetermine whether the requested storage item is located on the hostmachine (or otherwise associated with the file server VM orController/Service VM on the host machine). If so, the file server VMexecutes the requested operation. Otherwise, the file server VM respondsto the request with an indication that the requested storage item is noton the host machine, and may redirect the requesting user VM to the hostmachine on which the storage map indicates the storage item is located.The client may cache the address of the host machine on which thestorage item is located, so that the client may send subsequent requestsfor the storage item directly to that host machine.

In particular embodiments, the virtualized file server determines thelocation, e.g., host machine, at which to store a storage item such as afile or folder when the storage item is created. A file server VM mayattempt to create a file or folder using a Controller/Service VM on thesame host machine as the user VM that requested creation of the file, sothat the Controller/Service VM that controls access operations to thestorage item is co-located with the requesting user VM. In this way,file access operations between the user VM that is known to beassociated with the storage item and is thus likely to access thestorage item again (e.g., in the near future and/or on behalf of thesame user) may use local communication or short-distance communicationto improve performance, e.g., by reducing access times or increasingaccess throughput. Further, the virtualized file server may also attemptto store the storage item on a storage device that is local to theController/Service VM being used to create the storage item, so thatstorage access operations between the Controller/Service VM and thestorage device may use local or short-distance communication.

Further details of aspects, objects, and advantages of the invention aredescribed below in the detailed description, drawings, and claims. Boththe foregoing general description and the following detailed descriptionare exemplary and explanatory, and are not intended to be limiting as tothe scope of the invention. Particular embodiments may include all,some, or none of the components, elements, features, functions,operations, or steps of the embodiments disclosed above. The subjectmatter which can be claimed comprises not only the combinations offeatures as set out in the attached claims but also any othercombination of features in the claims, wherein each feature mentioned inthe claims can be combined with any other feature or combination ofother features in the claims. Furthermore, any of the embodiments andfeatures described or depicted herein can be claimed in a separate claimand/or in any combination with any embodiment or feature described ordepicted herein or with any of the features of the attached claims.

FIG. 1A illustrates a clustered virtualization environment according tosome particular embodiments. The architecture of FIG. 1A can beimplemented for a distributed platform that contains multiple hostmachines 100 a-c that manage multiple tiers of storage. The multipletiers of storage may include network-attached storage (NAS) that isaccessible through network 140, such as, by way of example and notlimitation, cloud storage 126, which may be accessible through theInternet, or local network-accessible storage 128 (e.g., a storage areanetwork (SAN)). Unlike the prior art, the present embodiment alsopermits local storage 122 that is within or directly attached to theserver and/or appliance to be managed as part of storage pool 160.Examples of such storage include Solid State Drives 125 (henceforth“SSDs”), Hard Disk Drives 127 (henceforth “HDDs” or “spindle drives”),optical disk drives, external drives (e.g., a storage device connectedto a host machine via a native drive interface or a direct attach serialinterface), or any other directly attached storage. These collectedstorage devices, both local and networked, form storage pool 160.Virtual disks (or “vDisks”) can be structured from the storage devicesin storage pool 160, as described in more detail below. As used herein,the term vDisk refers to the storage abstraction that is exposed by aController/Service VM (CVM) to be used by a user VM. In someembodiments, the vDisk is exposed via iSCSI (“internet small computersystem interface”) or NFS (“network file system”) and is mounted as avirtual disk on the user VM.

Each host machine 100 a-c runs virtualization software, such as VMWAREESX(I), MICROSOFT HYPER-V, or REDHAT KVM. The virtualization softwareincludes hypervisor 130 a-c to manage the interactions between theunderlying hardware and the one or more user VMs 101 a, 102 a, 101 b,102 b, 101 c, and 102 c that run client software. Though not depicted inFIG. 1A, a hypervisor may connect to network 140. In particularembodiments, a host machine 100 may be a physical hardware computingdevice; in particular embodiments, a host machine 100 may be a virtualmachine.

CVMs 110 a-c are used to manage storage and input/output (“I/O”)activities according to particular embodiments. These special VMs act asthe storage controller in the currently described architecture. Multiplesuch storage controllers may coordinate within a cluster to form aunified storage controller system. CVMs 110 may run as virtual machineson the various host machines 100, and work together to form adistributed system 110 that manages all the storage resources, includinglocal storage 122, networked storage 128, and cloud storage 126. TheCVMs may connect to network 140 directly, or via a hypervisor. Since theCVMs run independent of hypervisors 130 a-c, this means that the currentapproach can be used and implemented within any virtual machinearchitecture, since the CVMs can be used in conjunction with anyhypervisor from any virtualization vendor.

A host machine may be designated as a leader node within a cluster ofhost machines. For example, host machine 100 b, as indicated by theasterisks, may be a leader node. A leader node may have a softwarecomponent designated to perform operations of the leader. For example,CVM 110 b on host machine 100 b may be designated to perform suchoperations. A leader may be responsible for monitoring or handlingrequests from other host machines or software components on other hostmachines throughout the virtualized environment. If a leader fails, anew leader may be designated. In particular embodiments, a managementmodule (e.g., in the form of an agent) may be running on the leadernode.

Each CVM 110 a-c exports one or more block devices or NFS server targetsthat appear as disks to user VMs 101 and 102. These disks are virtual,since they are implemented by the software running inside CVMs 110 a-c.Thus, to user VMs 101 and 102, CVMs 110 a-c appear to be exporting aclustered storage appliance that contains some disks. All user data(including the operating system) in the user VMs 101 and 102 reside onthese virtual disks.

Significant performance advantages can be gained by allowing thevirtualization system to access and utilize local storage 122 asdisclosed herein. This is because I/O performance is typically muchfaster when performing access to local storage 122 as compared toperforming access to networked storage 128 across a network 140. Thisfaster performance for locally attached storage 122 can be increasedeven further by using certain types of optimized local storage devices,such as SSDs. Further details regarding methods and mechanisms forimplementing the virtualization environment illustrated in FIG. 1A aredescribed in U.S. Pat. No. 8,601,473, which is hereby incorporated byreference in its entirety.

FIG. 1B illustrates data flow within an example clustered virtualizationenvironment according to particular embodiments. As described above, oneor more user VMs and a CVM may run on each host machine 100 along with ahypervisor. As a user VM performs I/O operations (e.g., a read operationor a write operation), the I/O commands of the user VM may be sent tothe hypervisor that shares the same server as the user VM. For example,the hypervisor may present to the virtual machines an emulated storagecontroller, receive an I/O command and facilitate the performance of theI/O command (e.g., via interfacing with storage that is the object ofthe command, or passing the command to a service that will perform theI/O command). An emulated storage controller may facilitate I/Ooperations between a user VM and a vDisk. A vDisk may present to a userVM as one or more discrete storage drives, but each vDisk may correspondto any part of one or more drives within storage pool 160. Additionallyor alternatively, Controller/Service VM 110 a-c may present an emulatedstorage controller either to the hypervisor or to user VMs to facilitateI/O operations. CVMs 110 a-c may be connected to storage within storagepool 160. CVM 110 a may have the ability to perform I/O operations usinglocal storage 122 a within the same host machine 100 a, by connectingvia network 140 to cloud storage 126 or networked storage 128, or byconnecting via network 140 to local storage 122 b-c within another hostmachine 100 b-c (e.g., via connecting to another CVM 110 b or 110 c). Inparticular embodiments, any suitable computing system 700 may be used toimplement a host machine 100.

File System Architecture

FIG. 2A illustrates a clustered virtualization environment implementinga virtualized file server (VFS) 202 according to particular embodiments.In particular embodiments, the VFS 202 provides file services to uservirtual machines (user VMs) 101 and 102. The file services may includestoring and retrieving data persistently, reliably, and efficiently. Theuser virtual machines 101 and 102 may execute user processes, such asoffice applications or the like, on host machines 200 a-c. The storeddata may be represented as a set of storage items, such as filesorganized in a hierarchical structure of folders (also known asdirectories), which can contain files and other folders.

In particular embodiments, the VFS 202 may include a set of File ServerVirtual Machines (FSVMs) 170 a-c that execute on host machines 200 a-cand process storage item access operations requested by user VMs 200 a-cexecuting on the host machines 200 a-c. The FSVMs 170 a-c maycommunicate with storage controllers provided by CVMs 110 a-c executingon the host machines 200 a-c to store and retrieve files, folders, orother storage items on local storage 122 a-c associated with, e.g.,local to, the host machines 200 a-c. The network protocol used forcommunication between user VMs 101 and 102, FSVMs 170 a-c, and CVMs 110a-c via the network 140 may be Internet Small Computer Systems Interface(iSCSI), Server Message Block (SMB), Network File System (NFS), pNFS(Parallel NFS), or another appropriate protocol.

For the purposes of VFS 202, host machine 200 c may be designated as aleader node within a cluster of host machines. In this case, FSVM 170 con host machine 100 c may be designated to perform such operations. Aleader may be responsible for monitoring or handling requests from FSVMson other host machines throughout the virtualized environment. If FSVM170 c fails, a new leader may be designated for VFS 202.

In particular embodiments, the user VMs 101 and 102 may send data to theVFS 202 using write requests, and may receive data from it using readrequests. The read and write requests, and their associated parameters,data, and results, may be sent between a user VM 101 a and one or morefile server VMs (FSVMs) 170 a-c located on the same host machine as theuser VM 101 a or on different host machines from the user VM 101 a. Theread and write requests may be sent between host machines 200 a-c vianetwork 140, e.g., using a network communication protocol such as iSCSI,CIFS, SMB, TCP, IP, or the like. When a read or write request is sentbetween two VMs located on the same one of the host machines 200 a-c(e.g., between the user VM 101 a and the FSVM 170 a located on the hostmachine 200 a), the request may be sent using local communication withinthe host machine 200 a instead of via the network 140. As describedabove, such local communication may be substantially faster thancommunication via the network 140. The local communication may beperformed by, e.g., writing to and reading from shared memory accessibleby the user VM 101 a and the FSVM 170 a, sending and receiving data viaa local “loopback” network interface, local stream communication, or thelike.

In particular embodiments, the storage items stored by the VFS 202, suchas files and folders, may be distributed amongst multiple host machines200 a-c. In particular embodiments, when storage access requests arereceived from the user VMs 101 and 102, the VFS 202 identifies hostmachines 200 a-c at which requested storage items, e.g., folders, files,or portions thereof, are stored, and directs the user VMs 101 and 102 tothe locations of the storage items. The FSVMs 170 may maintain a storagemap, such as a sharding map 360 (shown in FIG. 3C), that maps names oridentifiers of storage items to their corresponding locations. Thestorage map may be a distributed data structure of which copies aremaintained at each FSVM 170 a-c and accessed using distributed locks orother storage item access operations. Alternatively, the storage map maybe maintained by a leader node such as the host machine 200 c, and theother host machines 200 a and 200 b may send requests to query andupdate the storage map to the leader host machine 200 c. Otherimplementations of the storage map are possible using appropriatetechniques to provide asynchronous data access to a shared resource bymultiple readers and writers. The storage map may map names oridentifiers of storage items in the form of text strings or numericidentifiers, such as folder names, files names, and/or identifiers ofportions of folders or files (e.g., numeric start offset positions andcounts in bytes or other units) to locations of the files, folders, orportions thereof. Locations may be represented as names of FSVMs 170a-c, e.g., “FSVM-1”, as network addresses of host machines 200 a-c onwhich FSVMs 170 a-c are located (e.g., “ip-addr1” or 128.1.1.10), or asother types of location identifiers.

When a user application executing in a user VM 101 a on one of the hostmachines 200 a initiates a storage access operation, such as reading orwriting data, the user VM 101 a may send the storage access operation ina request to one of the FSVMs 170 a-c on one of the host machines 200a-c. A FSVM executing on a host machine 200 b that receives a storageaccess request may use the storage map to determine whether therequested file or folder is located on the host machine 200 b (orotherwise associated with the FSVM 170 b or Controller/Service VM 110 bon the host machine 200 b). If the requested file or folder is locatedon the host machine 200 b (or otherwise associated with a VM on it), theFSVM 170 b executes the requested storage access operation. Otherwise,the FSVM 170 b responds to the request with an indication that the datais not on the host machine 200 b, and may redirect the requesting userVM 101 a to the host machine 200 c on which the storage map indicatesthe file or folder is located. The client may cache the address of thehost machine 200 c on which the file or folder is located, so that itmay send subsequent requests for the file or folder directly to the hostmachine 200 c.

As an example and not by way of limitation, the location of a file or afolder may be pinned to a particular host machine 200 a by sending afile service operation that creates the file or folder to a CVM 110 alocated on the particular host machine 200 a. The CVM 110 a subsequentlyprocesses file service commands for that file and sends correspondingstorage access operations to storage devices associated with the file.The CVM 110 a may associate local storage 122 a with the file if thereis sufficient free space on local storage 122 a. Alternatively, the CVM110 a may associate a storage device located on another host machine 200b, e.g., in local storage 122 b, with the file under certain conditions,e.g., if there is insufficient free space on the local storage 122 a, orif storage access operations between the CVM 110 a and the file areexpected to be infrequent. Files and folders, or portions thereof, mayalso be stored on other storage devices, such as the network-attachedstorage (NAS) 128 or the cloud storage 126 of the storage pool 160.

In particular embodiments, a name service 220, such as that specified bythe Domain Name System (DNS) Internet protocol, may communicate with thehost machines 200 a-c via the network 140 and may store a database ofdomain name (e.g., host name) to IP address mappings. The name service220 may be queried by the User VMs 101 to determine the IP address of aparticular host machine 200 a-c given a name of the host machine, e.g.,to determine the IP address of the host name ip-addr1 for the hostmachine 200 a. The name service 220 may be located on a separate servercomputer system or on one or more of the host machines 200. The namesand IP addresses of the host machines of the VFS instance 202, e.g., thehost machines 200, may be stored in the name service 220 so that theuser VMs 101 may determine the IP address of each of the host machines200. The name of each VFS instance 202, e.g., FS1, FS2, or the like, maybe stored in the name service 220 in association with a set of one ormore names that contains the name(s) of the host machines 200 of the VFSinstance 202. For example, the file server instance name FS1.domain.commay be associated with the host names ip-addr1, ip-addr2, and ip-addr3in the name service 220, so that a query of the name service 220 for theserver instance name “FS1” or “FS1.domain.com” returns the namesip-addr1, ip-addr2, and ip-addr3. Further, the name service 220 mayreturn the names in a different order for each name lookup request,e.g., using round-robin ordering, so that the sequence of names (oraddresses) returned by the name service for a file server instance nameis a different permutation for each query until all the permutationshave been returned in response to requests, at which point thepermutation cycle starts again, e.g., with the first permutation. Inthis way, storage access requests from user VMs 101 may be balancedacross the host machines 200, since the user VMs 101 submit requests tothe name service 220 for the address of the VFS instance 202 for storageitems for which the user VMs 101 do not have a record or cache entry, asdescribed below.

In particular embodiments, each FSVM 170 may have two IP addresses: anexternal IP address and an internal IP address. The external IPaddresses may be used by SMB/CIFS clients, such as user VMs 101, toconnect to the FSVMs 170. The external IP addresses may be stored in thename service 220. The IP addresses ip-addr1, ip-addr2, and ip-addr3described above are examples of external IP addresses. The internal IPaddresses may be used for iSCSI communication to CVMs 110, e.g., betweenthe FSVMs 170 and the CVMs 110, and for communication between the CVMs110 and storage devices in the storage pool 160. Other internalcommunications may be sent via the internal IP addresses as well, e.g.,file server configuration information may be sent from the CVMs 110 tothe FSVMs 170 using the internal IP addresses, and the CVMs 110 may getfile server statistics from the FSVMs 170 via internal communication asneeded.

Since the VFS 202 is provided by a distributed set of FSVMs 170 a-c, theuser VMs 101 and 102 that access particular requested storage items,such as files or folders, do not necessarily know the locations of therequested storage items when the request is received. A distributed filesystem protocol, e.g., MICROSOFT DFS or the like, is therefore used, inwhich a user VM 101 a may request the addresses of FSVMs 170 a-c from aname service 220 (e.g., DNS). The name service may send one or morenetwork addresses of FSVMs 170 a-c to the user VM 101 a, in an orderthat changes for each subsequent request. These network addresses arenot necessarily the addresses of the FSVM 170 b on which the storageitem requested by the user VM 101 a is located, since the name service220 does not necessarily have information about the mapping betweenstorage items and FSVMs 170 a-c. Next, the user VM 170 a may send anaccess request to one of the network addresses provided by the nameservice, e.g., the address of FSVM 170 b. The FSVM 170 b may receive theaccess request and determine whether the storage item identified by therequest is located on the FSVM 170 b. If so, the FSVM 170 b may processthe request and send the results to the requesting user VM 101 a.However, if the identified storage item is located on a different FSVM170 c, then the FSVM 170 b may redirect the user VM 101 a to the FSVM170 c on which the requested storage item is located by sending a“redirect” response referencing FSVM 170 c to the user VM 101 a. Theuser VM 101 a may then send the access request to FSVM 170 c, which mayperform the requested operation for the identified storage item.

A particular VFS 202, including the items it stores, e.g., files andfolders, may be referred to herein as a VFS “instance” 202 and may havean associated name, e.g., FS1, as described above. Although a VFSinstance 202 may have multiple FSVMs distributed across different hostmachines 200, with different files being stored on different hostmachines 200, the VFS instance 202 may present a single name space toits clients such as the user VMs 101. The single name space may include,for example, a set of named “shares” and each share may have anassociated folder hierarchy in which files are stored. Storage itemssuch as files and folders may have associated names and metadata such aspermissions, access control information, size quota limits, file types,files sizes, and so on. As another example, the name space may be asingle folder hierarchy, e.g., a single root directory that containsfiles and other folders. User VMs 101 may access the data stored on adistributed VFS instance 202 via storage access operations, such asoperations to list folders and files in a specified folder, create a newfile or folder, open an existing file for reading or writing, and readdata from or write data to a file, as well as storage item manipulationoperations to rename, delete, copy, or get details, such as metadata, offiles or folders. Note that folders may also be referred to herein as“directories.”

In particular embodiments, storage items such as files and folders in afile server namespace may be accessed by clients such as user VMs 101 byname, e.g., “\Folder-1\File-1” and “\Folder-2\File-2” for two differentfiles named File-1 and File-2 in the folders Folder-1 and Folder-2,respectively (where Folder-1 and Folder-2 are sub-folders of the rootfolder). Names that identify files in the namespace using folder namesand file names may be referred to as “path names.” Client systems mayaccess the storage items stored on the VFS instance 202 by specifyingthe file names or path names, e.g., the path name “\Folder-1 \File-1”,in storage access operations. If the storage items are stored on a share(e.g., a shared drive), then the share name may be used to access thestorage items, e.g., via the path name “\\Share-1\Folder-1 \File-1” toaccess File-1 in folder Folder-1 on a share named Share-1.

In particular embodiments, although the VFS instance 202 may storedifferent folders, files, or portions thereof at different locations,e.g., on different host machines 200, the use of different host machinesor other elements of storage pool 160 to store the folders and files maybe hidden from the accessing clients. The share name is not necessarilya name of a location such as a host machine 200. For example, the nameShare-1 does not identify a particular host machine 200 on which storageitems of the share are located. The share Share-1 may have portions ofstorage items stored on three host machines 200 a-c, but a user maysimply access Share-1, e.g., by mapping Share-1 to a client computer, togain access to the storage items on Share-1 as if they were located onthe client computer. Names of storage items, such as file names andfolder names, are similarly location-independent. Thus, although storageitems, such as files and their containing folders and shares, may bestored at different locations, such as different host machines 200 a-c,the files may be accessed in a location-transparent manner by clients(such as the user VMs 101 and 102). Thus, users at client systems neednot specify or know the locations of each storage item being accessed.The VFS 202 may automatically map the file names, folder names, or fullpath names to the locations at which the storage items are stored. As anexample and not by way of limitation, a storage item's physical locationmay be specified by the name or address of the host machine 200 a-c onwhich the storage item is located, the name, address, or identity of theFSVM 170 a-c that provides access to the storage item on the hostmachine 200 a-c on which the storage item is located, the particulardevice (e.g., SSD or HDD) of the local storage 122 a (or other type ofstorage in storage pool 160) on which the storage item is located, andthe address on the device, e.g., disk block numbers. A storage item suchas a file may be divided into multiple parts that may be located ondifferent host machines 200 a-c, in which case access requests for aparticular portion of the file may be automatically mapped to thelocation of the portion of the file based on the portion of the filebeing accessed (e.g., the offset from the beginning of the file and thenumber of bytes being accessed).

In particular embodiments, VFS 202 determines the location, e.g.,particular host machine 200 a-c, at which to store a storage item whenthe storage item is created. For example, a FSVM 170 a may attempt tocreate a file or folder using a Controller/Service VM 110 a on the samehost machine 200 a as the user VM 101 a that requested creation of thefile, so that the Controller/Service VM 110 a that controls accessoperations to the file folder is co-located with the user VM 101 a. Inthis way, since the user VM 101 a is known to be associated with thefile or folder and is thus likely to access the file again, e.g., in thenear future or on behalf of the same user, access operations may uselocal communication or short-distance communication to improveperformance, e.g., by reducing access times or increasing accessthroughput. If there is a local CVM 110 a on the same host machine asthe FSVM 170 a, the FSVM 170 a may identify it and use it by default. Ifthere is no local CVM 110 a on the same host machine as the FSVM 170 a,a delay may be incurred for communication between the FSVM 170 a and aCVM 110 b on a different host machine 200 b. Further, the VFS 202 mayalso attempt to store the file on a storage device that is local to theCVM 110 a being used to create the file, such as local storage 122 a, sothat storage access operations between the CVM 110 a and local storage122 a may use local or short-distance communication.

In particular embodiments, if a CVM 110 a is unable to store the storageitem in local storage 122 a, e.g., because local storage 122 a does nothave sufficient available free space, then the file may be stored inlocal storage 122 b of a different host machine 200 b. In this case, thestored file is not physically local to the host machine 200 a, butstorage access operations for the file are performed by thelocally-associated CVM 110 a and FSVM 170 a, and the CVM 110 a maycommunicate with local storage 122 b on the remote host machine 200 busing a network file sharing protocol, e.g., iSCSI, SAMBA or the like.

In particular embodiments, if a virtual machine, such as a user VM 101a, CVM 110 a, or FSVM 170 a, moves from a host machine 200 a to adestination host machine 200 b, e.g., because of resource availabilitychanges, and data items such as files or folders associated with the VMare not locally accessible on the destination host machine 200 b, thendata migration may be performed for the data items associated with themoved VM to migrate them to the new host machine 200 b, so that they arelocal to the moved VM on the new host machine 200 b. FSVMs 170 maydetect removal and addition of CVMs 110 (as may occur, for example, whena CVM 110 fails or is shut down) via the iSCSI protocol or othertechnique, such as heartbeat messages. As another example, a FSVM 170may determine that a particular file's location is to be changed, e.g.,because a disk on which the file is stored is becoming full, becausechanging the file's location is likely to reduce network communicationdelays and therefore improve performance, or for other reasons. Upondetermining that a file is to be moved, VFS 202 may change the locationof the file by, for example, copying the file from its existinglocation(s), such as local storage 122 a of a host machine 200 a, to itsnew location(s), such as local storage 122 b of host machine 200 b (andto or from other host machines, such as local storage 122 c of hostmachine 200 c if appropriate), and deleting the file from its existinglocation(s). Write operations on the file may be blocked or queued whilethe file is being copied, so that the copy is consistent. The VFS 202may also redirect storage access requests for the file from an FSVM 170a at the file's existing location to a FSVM 170 b at the file's newlocation.

In particular embodiments, VFS 202 includes at least three File ServerVirtual Machines (FSVMs) 170 a-c located on three respective hostmachines 200 a-c. To provide high-availability, there may be a maximumof one FSVM 170 a for a particular VFS instance 202 per host machine 200in a cluster. If two FSVMs 170 are detected on a single host machine200, then one of the FSVMs 170 may be moved to another host machineautomatically, or the user (e.g., system administrator) may be notifiedto move the FSVM 170 to another host machine. The user may move a FSVM170 to another host machine using an administrative interface thatprovides commands for starting, stopping, and moving FSVMs 170 betweenhost machines 200.

In particular embodiments, two FSVMs 170 of different VFS instances 202may reside on the same host machine 200 a. If the host machine 200 afails, the FSVMs 170 on the host machine 200 a become unavailable, atleast until the host machine 200 a recovers. Thus, if there is at mostone FSVM 170 for each VFS instance 202 on each host machine 200 a, thenat most one of the FSVMs 170 may be lost per VFS 202 per failed hostmachine 200. As an example, if more than one FSVM 170 for a particularVFS instance 202 were to reside on a host machine 200 a, and the VFSinstance 202 includes three host machines 200 a-c and three FSVMs 170,then loss of one host machine would result in loss of two-thirds of theFSVMs 170 for the VFS instance 202, which would be more disruptive andmore difficult to recover from than loss of one-third of the FSVMs 170for the VFS instance 202.

In particular embodiments, users, such as system administrators or otherusers of the user VMs 101, 102, may expand the cluster of FSVMs 170 byadding additional FSVMs 170. Each FSVM 170 a may be associated with atleast one network address, such as an IP (Internet Protocol) address ofthe host machine 200 a on which the FSVM 170 a resides. There may bemultiple clusters, and all FSVMs of a particular VFS instance areordinarily in the same cluster. The VFS instance 202 may be a member ofa MICROSOFT ACTIVE DIRECTORY domain, which may provide authenticationand other services such as name service 220.

FIG. 2B illustrates data flow within a clustered virtualizationenvironment implementing a VFS instance 202 in which stored items suchas files and folders used by user VMs 101 are stored locally on the samehost machines 200 as the user VMs 101 according to particularembodiments. As described above, one or more user VMs 101 and aController/Service VM 110 may run on each host machine 200 along with ahypervisor 130. As a user VM 101 processes I/O commands (e.g., a read orwrite operation), the I/O commands may be sent to the hypervisor 130 onthe same server or host machine 200 as the user VM 101. For example, thehypervisor 130 may present to the user VMs 101 a VFS instance 202,receive an I/O command, and facilitate the performance of the I/Ocommand by passing the command to a FSVM 170 that performs the operationspecified by the command. The VFS 202 may facilitate I/O operationsbetween a user VM 101 and a virtualized file system. The virtualizedfile system may appear to the user VM 101 as a namespace of mappableshared drives or mountable network file systems of files anddirectories. The namespace of the virtualized file system may beimplemented using storage devices in the local storage 122, such asdisks 204, onto which the shared drives or network file systems, files,and folders, or portions thereof, may be distributed as determined bythe FSVMs 170. The VFS 202 may thus provide features disclosed herein,such as efficient use of the disks 204, high availability, scalability,and others. The implementation of these features may be transparent tothe user VMs 101, 102. The FSVMs 170 may present the storage capacity ofthe disks 204 of the host machines 200 as an efficient,highly-available, and scalable namespace in which the user VMs 101, 102may create and access shares, files, folders, and the like.

As an example, a network share may be presented to a user VM 101 as oneor more discrete virtual disks, but each virtual disk may correspond toany part of one or more virtual or physical disks 204 within storagepool 160. Additionally or alternatively, the FSVMs 170 may present a VFS202 either to the hypervisor 130 or to user VMs 101 of a host machine200 to facilitate I/O operations. The FSVMs 170 may access the localstorage 122 via Controller/Service VMs 110. As described above withreference to FIG. 1B, a Controller/Service VM 110 a may have the abilityto perform I/O operations using local storage 122 a within the same hostmachine 200 a by connecting via the network 140 to cloud storage 126 ornetworked storage 128, or by connecting via the network 140 to localstorage 122 b-c within another host machine 200 b-c (e.g., by connectingto another Controller/Service VM 110 b-c).

In particular embodiments, each user VM 101 may access one or morevirtual disk images 206 stored on one or more disks 204 of the localstorage 122, the cloud storage 126, and/or the networked storage 128.The virtual disk images 206 may contain data used by the user VMs 101,such as operating system images, application software, and user data,e.g., user home folders and user profile folders. For example, FIG. 2Billustrates three virtual machine images 206 a-c. The virtual machineimage 206 a may be a file named UserVM101 a.vmdisk (or the like) storedon disk 204 a of local storage 122 a of host machine 200 a. The virtualmachine image 206 a may store the contents of the user VM 101 a's harddrive. The disk 204 a on which the virtual machine image 206 a is “localto” the user VM 101 a on host machine 200 a because the disk 204 a is inlocal storage 122 a of the host machine 200 a on which the user VM 101 ais located. Thus, the user VM 101 a may use local (intra-host machine)communication to access the virtual machine image 206 a moreefficiently, e.g., with less latency and higher throughput, than wouldbe the case if the virtual machine image 206 a were stored on disk 204 bof local storage 122 b of a different host machine 200 b, becauseinter-host machine communication across the network 140 would be used inthe latter case. Local communication within a host machine 200 a isdescribed in further detail with reference to FIG. 4C. Similarly, avirtual machine image 206 b, which may be a file named UserVM101b.vmdisk (or the like), is stored on disk 204 b of local storage 122 bof host machine 200 b, and the image 206 b is local to the user VM 101 blocated on host machine 200 b. Thus, the user VM 101 a may access thevirtual machine image 206 b more efficiently than the virtual machine206 a on host machine 200 a, for example. In another example, the CVM110 c may be located on the same host machine 200 c as the user VM 101 cthat accesses a virtual machine image 206 c (UserVM101 c.vmdisk) of theuser VM 101 c, with the virtual machine image file 206 c being stored ona different host machine 200 b than the user VM 101 c and the CVM 110 c.In this example, communication between the user VM 101 c and the CVM 110c may still be local, e.g., more efficient than communication betweenthe user VM 101 c and a CVM 110 b on a different host machine 200 b, butcommunication between the CVM 110 c and the disk 204 b on which thevirtual machine image 206 c is stored is via the network 140, as shownby the dashed lines between CVM 110 c and the network 140 and betweenthe network 140 and local storage 122 b. The communication between CVM110 c and the disk 204 b is not local, and thus may be less efficientthan local communication such as may occur between the CVM 110 c and adisk 204 c in local storage 122 c of host machine 200 c. Further, a userVM 101 c on host machine 200 c may access data such as the virtual diskimage 206 c stored on a remote (e.g., non-local) disk 204 b via networkcommunication with a CVM 110 b located on the remote host machine 200 b.This case may occur if CVM 110 c is not present on host machine 200 c,e.g., because CVM 110 c has failed, or if the FSVM 170 c has beenconfigured to communicate with local storage 122 b on host machine 200 bvia the CVM 110 b on host machine 200 b, e.g., to reduce computationalload on host machine 200 c.

In particular embodiments, since local communication is expected to bemore efficient than remote communication, the FSVMs 170 may storestorage items, such as files or folders, e.g., the virtual disk images206, on local storage 122 of the host machine 200 on which the user VM101 that is expected to access the files is located. A user VM 101 maybe expected to access particular storage items if, for example, thestorage items are associated with the user VM 101, such as byconfiguration information. For example, the virtual disk image 206 a maybe associated with the user VM 101 a by configuration information of theuser VM 101 a. Storage items may also be associated with a user VM 101via the identity of a user of the user VM 101. For example, files andfolders owned by the same user ID as the user who is logged into theuser VM 101 a may be associated with the user VM 101 a. If the storageitems expected to be accessed by a user VM 101 a are not stored on thesame host machine 200 a as the user VM 101 a, e.g., because ofinsufficient available storage capacity in local storage 122 a of thehost machine 200 a, or because the storage items are expected to beaccessed to a greater degree (e.g., more frequently or by more users) bya user VM 101 b on a different host machine 200 b, then the user VM 101a may still communicate with a local CVM 110 a to access the storageitems located on the remote host machine 200 b, and the local CVM 110 amay communicate with local storage 122 b on the remote host machine 200b to access the storage items located on the remote host machine 200 b.If the user VM 101 a on a host machine 200 a does not or cannot use alocal CVM 110 a to access the storage items located on the remote hostmachine 200 b, e.g., because the local CVM 110 a has crashed or the userVM 101 a has been configured to use a remote CVM 110 b, thencommunication between the user VM 101 a and local storage 122 b on whichthe storage items are stored may be via a remote CVM 110 b using thenetwork 140, and the remote CVM 110 b may access local storage 122 busing local communication on host machine 200 b. As another example, auser VM 101 a on a host machine 200 a may access storage items locatedon a disk 204 c of local storage 122 c on another host machine 200 c viaa CVM 110 b on an intermediary host machine 200 b using networkcommunication between the host machines 200 a and 200 b and between thehost machines 200 b and 200 c.

One of the benefits of a virtualized architecture is the ease ofhandling increases in load by “scaling up” virtual machines that satisfya resource usage threshold (e.g., 95% of CPU or memory usage) byallocating additional resources (e.g., CPU or memory). This techniquemay have its limitations, however, since virtualized resources areultimately limited by the underlying physical resources. Particularembodiments may augment this technique by “scaling out” once a lowerthreshold is satisfied (e.g., 75%): creating one or more new FSVMs (andmoving one or more volume groups (or any other minimum moveable unit)from heavily loaded FSVMs to being served by the newly created FSVM.Particular embodiments may conversely “scale down” and/or “scale in”virtual machines in accordance with a resource usage threshold, clusterrules governing a minimum or maximum number of nodes to be permitted inthe cluster, cluster rules governing a minimum or maximum number of noderesources to be permitted in the cluster, or any other applicable rulegoverning the number of VFS elements to be permitted in the cluster.

However, when resource consumption for each FSVM 170 is provided in avolume group-agnostic fashion (e.g., reported on for each SMBconnection), determining which volume group to move may not bestraightforward. In particular embodiments, rather than preciselydetermining the resource consumption that can be attributed to each ofthe volume groups based on access log data, the load may be approximatedby a heuristic based on: assessing I/O statistics that are available ona volume-group basis (e.g., read or write operations served by MBps),summing the statistics for each of the FSVMs, determining the percentageof I/O operations handled by each of the FSVMs 170 compared to the totalI/O operations across all FSVMs 170 in VFS 202, and then apply thedetermined percentage to the total resources consumed by all SMBconnections to VFS 202 in order to determine the resources consumed byeach of the FSVMs 170. Once the resources consumed by a given FSVM havebeen determined, VFS 202 may assess whether the FSVM is close tocapacity by determining whether the estimated load satisfies (e.g.,meets or exceeds) a capacity threshold for the FSVM. If yes, then VFS202 may add another host machine to the VFS cluster, create a new FSVMon the newly added host machine, and move one or more volume groups fromthe overloaded FSVM to the new FSVM.

In one example, VFS 202 may include three FSVMs, each of which arehandling I/O requests for three volume groups. In particularembodiments, the total resource usage across all three FSVMs may bedistributed across the nine volume groups and attributed to each volumegroup in proportion with available statistics on read operations servedby MBps, as shown in the table below:

Volume Reads % of Load Load Load FSVM Group (MBps) (by VG) (% of VFS) (%of max) FSVM₁ VG₁ 127 12.7%  39.1% 88.1% VG₂ 118 11.8%  VG₃ 72 7.2%FSVM₂ VG₄ 22 2.2% 34.6% 78.8% VG₅ 170 17.0%  VG₆ 88 8.8% FSVM₃ VG₇ 939.3% 26.3% 59.2% VG₈ 62 6.2% VG₉ 58 5.8%

Once VFS 202 determines that the volume groups being served by FSVM₁ areresponsible for serving 317 MBps of read operations, VFS 202 calculatesthat FSVM₁ is serving 39.1% of the total read operations served acrossall FSVMs in VFS 202 (e.g., 810 MBps). VFS 202 then allocates a portionof the total CPU usage across all SMB connections (e.g., 4.5 billionoperations per second) by applying the calculated percentage (39.1%) tothe total CPU usage, which results in an allocation of 1.76 billionoperations per second of CPU usage for FSVM₁. VFS 202 can now computethe ratio of the allocated load (1.76 billion operations per second) tothe maximum load capacity (e.g., 2 billion operations per second) of thehost machine running FSVM₁—the allocated load is computed to beutilizing 88% of the maximum capacity for the host machine's CPU.

Similarly, VFS 202 determines that the volume groups being served byFSVM₂ are responsible for serving 280 MBps of read operations. VFS 202then calculates that FSVM₂ is serving 34.6% of the total read operationsserved across all FSVMs in VFS 202. VFS 202 then allocates a portion ofthe total CPU usage across all SMB connections by applying thecalculated percentage to the total CPU usage, which results in anallocation of 1.56 billion operations per second of CPU usage for FSVM₁.VFS 202 can now compute the ratio of the allocated load to the maximumload capacity (e.g., 2 billion operations per second) of the hostmachine running FSVM₂—the allocated load is computed to be utilizing 78%of the maximum capacity for the host machine's CPU.

VFS 202 subsequently allocates a load of 1.18 billion operations persecond to FSVM₃ (serving 26.3% of the total read operations servedacross all FSVMs), which is computed to be utilizing 59% of the maximumload capacity (e.g., 2 billion operations per second) of the hostmachine running FSVM₃.

In an example where the minimum threshold for moving a volume group to anew FSVM is 80%, one FSVM (FSVM₁) is determined to have a utilizationsatisfying the threshold. VFS 202 may add a new host machine to the VFScluster, create a new FSVM on the newly added host machine (e.g.,FSVM₄), and then select one or more volume groups to move from the FSVMserving the heaviest load of I/O operations (FSVM₁) to newly createdFSVM₄, such as the most-accessed volume group (VG₁):

Volume Reads % of Load % of Load % of Load FSVM Group (MBps) (by VG) (byFSVM) (by FSVM) FSVM₁ VG₂ 118 11.8%  23.5% 52.8% VG₃ 72 7.2% FSVM₂ VG₄22 2.2% 34.6% 77.8% VG₅ 170 17.0%  VG₆ 88 8.8% FSVM₃ VG₇ 93 9.3% 26.3%59.2% VG₈ 62 6.2% VG₉ 58 5.8% FSVM₄ VG₁ 127 12.7%  15.7% 35.3%By moving VG₁ to be served by FSVM₄, FSVM₁ now serves 317 MBps of readoperations (23.5% of the total read operations across all FSVMs), whichdrops its allocation of the total CPU usage to 1.056 billion operationsper second—at 52.8% of maximum capacity, FSVM₁ is now well-positioned tohandle future increases in load.

In an example where the minimum threshold for moving a volume group to anew FSVM is 75%, computed utilization for both FSVM₁ and FSVM₂ (88% and78%, respectively) satisfies (e.g., meets or exceeds) the minimumthreshold for moving a volume group to a new FSVM. VFS 202 may thuscreate one or more new FSVMs, and then select one or more volume groupsto move to the one or more new FSVMs.

In some embodiments, VFS 202 may simply add two new host machines to thecluster (one for each of the overloaded FSVMs) and then moves themost-accessed volume group from each of the overloaded FSVMs (VG₁ onFSVM₁ and VG₅ on FSVM₂) to be served by a new FSVM on a respective oneof the two new host machines:

Volume Reads % of Load % of Load % of Load FSVM Group (MBps) (by VG) (byFSVM) (by FSVM) FSVM₁ VG₂ 118 11.8%  23.5% 52.8% VG₃ 72 7.2% FSVM₂ VG₄22 2.2% 13.6% 30.6% VG₆ 88 8.8% FSVM₃ VG₇ 93 9.3% 26.3% 59.2% VG₈ 626.2% VG₉ 58 5.8% FSVM₄ VG₁ 127 12.7%  15.7% 35.3% FSVM₅ VG₅ 170 17.0% 21.0% 47.2%

By moving VG₅ to be served by FSVM₅, FSVM₂ now serves only 110 MBps ofread operations (13.6% of the total read operations across all FSVMs),which drops its allocation of the total CPU usage to 0.61 billionoperations per second—at 30.6% of maximum capacity, FSVM₂ is nowwell-positioned to handle future increases in load.

In some embodiments, VFS 202 may determine whether it may be sufficientto add only one new host machine to the cluster and then move one ormore volume groups from each of the overloaded FSVMs (FSVM₁ and FSVM₂)to be served by a new FSVM created on the new host machine:

Volume Reads % of Load % of Load % of Load FSVM Group (MBps) (by VG) (byFSVM) (by FSVM) FSVM₁ VG₁ 127 12.7%  30.2% 68.1% VG₂ 118 11.8%  FSVM₂VG₄ 22 2.2% 23.7% 53.3% VG₅ 170 17.0%  FSVM₃ VG₇ 93 9.3% 26.3% 59.2% VG₈62 6.2% VG₉ 58 5.8% FSVM₄ VG₃ 72 7.2% 19.8% 44.4% VG₆ 88 8.8%By moving VG₃ and VG₆ to be served by FSVM₄, FSVM₁ now serves 245 MBpsof read operations (30.2% of the total read operations across allFSVMs), which drops its allocation of the total CPU usage to 1.36billion operations per second, and FSVM₂ now serves 192 MBps of readoperations (23.7% of the total read operations across all FSVMs), whichdrops its allocation of the total CPU usage to 1.07 billion operationsper second. At 68.1% and 53.3% of maximum capacity respectively, FSVM₁and FSVM₂ are now well-positioned to handle future increases in load, asis FSVM₄ at 44.4% of its maximum capacity.

In particular embodiments, VFS 202 may only review historical loadstatistics for the volume groups within a specified duration of time(e.g., within a two-hour rolling window). In particular embodiments, VFS202 may review historical load statistics for the volume groups andre-distribute load on a periodic basis (e.g., every 24 hours); inparticular embodiments, the specified duration of time may have a directcorrelation with the periodic basis (e.g., wherein the specifiedduration is 50% or 100% of the periodic basis).

In particular embodiments, VFS 202 may review multiple types ofhistorical load statistics for the volume groups (e.g., both readoperations and write operations). In making the determination whetherthe load distribution exceeds a threshold and should be re-distributed,VFS 202 may apply weights to different types of historical loadstatistics (e.g., weighting write operations more heavily than readoperations). In particular embodiments, VFS 202 may pin certain volumegroups based on type of historical load statistics (e.g., volume groupshandling a high percentage of write operations should not be moved ifpossible).

In particular embodiments, VFS 202 may decide whether one or more volumegroups may be moved based on policies (e.g., security policies orregulatory policies) or prioritization (e.g., don't move volume groupswith low I/O latency, or don't move volume groups housing datadesignated as mission-critical).

In particular embodiments, VFS 202 may incorporate a machine-learningmodel trained on historical data to predict which volume groups may needto be moved, thereby facilitating pre-emptive movement of volume groupsduring lightly loaded periods of time.

In particular embodiments, a system (VFS 202) for managing data accessusing a virtualized file server may comprise (1) a plurality of hostmachines implementing a virtualization environment, wherein each of thehost machines comprises a hypervisor and at least one user virtualmachine (user VM); and (2) a virtualized file server comprising aplurality of file server virtual machines (FSVMs) and a storage poolcomprising a plurality of volume groups, wherein each of the FSVMs isrunning on one of a subset of the host machines, and wherein each of theFSVMs conduct I/O transactions with their assigned volume groups in thestorage pool. Periodically, the system may review I/O statistics for thevolume groups and correlate a percentage of I/O operations being handledby each of the FSVMs to resource usage by each of the FSVMs. Based onthe correlated percentage, the system may identify one or moreoverloaded FSVMs and determine how many new host machines to add to thesubset. Finally, the system may assign one or more of the volume groupsto one or more new host machines in the subset.

Particular embodiments may rebalance data amongst FSVM nodes in acluster immediately following a scale-out or scale-in VFS operation byreassigning volume groups. For example, during a scale-in VFS operation,orphan volume groups formerly assigned to a FSVM that no longer existsmay be reassigned to one of the remaining FSVMs in the cluster. Inanother example, during a scale-out VFS operation, volume groups may bemoved to newly created FSVMs in the cluster. In either example,particular embodiments may further move other volume groups around inorder to more evenly distribute load amongst the current set of FSVMs.Selection of which volume groups to move and selection of the FSVMs towhich the volume groups will be moved may be accomplished based onanalysis of load statistics as described above. In particularembodiments, name service information and DNS information may also beupdated for volume groups that are moved. In particular embodiments,volume group reassignments may be rolled back in the case of failure toadd or remove FSVMs or failure to move one or more volume groups.

Conventionally, NAS 128 comprises one or more dedicated physical storagedevices that must be physically attached to network 140 and configured.In particular embodiments, NAS 128 may be provided in the form of one ormore existing host machines 200 running CVMs 110. Particular embodimentsof an administration tool may install and deploy VFS 202 (includingFSVMs 170) to run as a service on top of NAS 128. If a prior version ofVFS 202 is already deployed, the administration tool may simply upgradeFSVMs 170. Particular embodiments of the administration tool mayautomatically upgrade FSVMs 170 upon receiving a notification regardingavailability of upgraded software for installation. Upon completingdownload of the installer software, the administration tool mayautomatically detect and resolve incompatibilities between the CVMsoftware and VFS 202 (e.g., by upgrading the CVM software) beforeinstalling VFS 202, and then finally deploy VFS 202. Particularembodiments of the administration tool may provide a list of one or moredownloadable versions of installer software for VFS 202 at a portal,where the user may select one of the versions and commence the download.

In particular embodiments, the administration tool may handleinstallation, deployment, and upgrade of other elements of theinfrastructure depicted in FIGS. 2A-2B, such as, by way of example andnot limitation, CVMs 110, user VMs 101 and 102, or applications runningon the user VMs.

Examples described herein may include one or more file system virtualmachine managers (FSVM managers) which may facilitate installation offile server VMs described herein, setup of file server VMs describedherein, configuration of file server VMs described herein, orcombinations thereof.

FIG. 8 is a schematic illustration of a system including a controller VMhaving a file system virtual machine manager (FSVM manager) arranged inaccordance with embodiments described herein. The system of FIG. 8includes controller VM 110A, admin system 808 and portal 812. Thecontroller VM 110A may include FSVM manager 804 and FSVM config data806. The admin system 808 may host user interface 810. The portal 812may store FSVM image 814. Additional, fewer, and/or different componentsmay be used in other examples. The controller VM 110A refers to thecontroller/service VM 110 a of FIG. 2A and FIG. 2B and described herein.Other controller VMs may be used to implement the controller VM 110A inother examples.

All or combinations of components shown in FIG. 2A and FIG. 2B may bepresent in the system of FIG. 8, but are not shown in FIG. 8 for clarityof the components shown. For example, the system of FIG. 8 may include aplurality of host machines (e.g., host machines 200 a-c of FIG. 2A,which may also be referred to as computing nodes). The plurality of hostmachines may implement a virtualization environment. Each of the hostmachines may include a hypervisor (e.g., hypervisors 130 a-c of FIG.2A). Note that, in some examples, the hypervisors 130 a-c may all be ofa same type. However, in other example, at least one of the hypervisors130 a-c may be of a different type than at least one other one of thehypervisors 130 a-c. Examples of types of hypervisors include VMWARE,ESX(I), MICROSOFT HYPER-V, and REDHAT KVM.

The system of FIG. 8 may include one or more controller VMs (e.g.,controller/service VMs 110 a-c of FIG. 2A). The controller/service VMsmay perform as described herein. The system of FIG. 8 may include adistributed file server (e.g., the virtualized file server 202 of FIG.2A) which may include a plurality of FSVMs (e.g., FSVMs 170 a-c of FIG.2A) and a storage pool (e.g., storage pool 160 of FIG. 2A). Each of theFSVMs may be running on one of the host machines. The FSVMs may conductI/O transactions with the storage pool as described herein.

Controller VMs described herein (e.g., controller/service VMs) mayinclude an FSVM manager, such as the FSVM manager 804 included incontroller VM 110A of FIG. 8. Generally, a single FSVM manager may beprovided in a controller VM. The FSVM manager may in some examplescontrol setup, installation, change, and/or configuration of multiple(e.g., all) FSVMs in a virtualized file server described herein. Forexample, the FSVM manager 804 of controller VM 110A may control setup,installation, change, and/or configuration of FSVM 170 a, FSVM 170 b,and FSVM 170 c of FIG. 2A and FIG. 2B. For example, a lead CVM in avirtualized environment (e.g., in a cluster) may host an FSVM managerfor the virtualized environment (e.g., one FSVM manager for thecluster). In other example, multiple FSVM managers may be provided in acluster. The FSVM manager may generally be implemented using software(e.g., using executable instructions encoded in computer readable mediafor performing FSVM manager functions described herein). The FSVMmanager may be provided by a controller VM (e.g., may be a service onthe CVM). Accordingly, a CVM may be referred to as performing theactions, or being configured to perform the actions, described withrespect to an FSVM manager herein.

Controller VMs described herein (e.g., controller/service VMs) mayinclude file server virtual machine configuration data. For example, thecontroller VM 110A may include (e.g., store) FSVM config data 806. TheFSVM configuration data may be stored on memory associated with thecontroller VM 110A and/or FSVM manager 804. The FSVM configuration datagenerally includes associations between user-specified target inputs andresource allocations for FSVMs to achieve the user-specified targetinputs. For example, the FSVM config data 806 may include associationsbetween inputs (e.g., target number of connections and targetthroughput) and resulting FSVM configurations to achieve the desiredinput (e.g., a number of FSVMs and/or memory and/or compute resourceallocations for the FSVMs). In some examples, the FSVM configurationdata may be based, for example, on test data from virtualizedenvironments. For example, a virtualized environment may be tested witha variety of number of FSVMs, and memory and compute resourceallocations for the FSVMs. Performance metrics for the test system—e.g.,number of connections and/or throughput of the FSVMs may be recorded.The recorded performance may be associated with the corresponding numberof FSVMs and memory and compute resource allocations and stored as FSVMconfig data 806.

The number of FSVMs generally refers to the number of FSVMs used in avirtualized file server—e.g., the virtualized file server 202 shown inFIG. 2A is illustrated as including three FSVMs—FSVM 170 a, FSVM 170 b,and FSVM 170 c. Any number of FSVMs may generally be used including 1,2, 3, 4, 5, 6, 7, 8, 9, 10 FSVMs, or more FSVMs in some examples.

The memory allocation for the FSVMs generally refers to the amount ofmemory allocated to the FSVM. In some examples, a same amount of memorymay be allocated to each FSVM in a virtualized file server. In someexamples, different amounts of memory may be allocated to differentFSVMs in a virtualized file server. For example, each of the file serverVMs shown in FIG. 2A may have a same amount of memory allocated to it.In other examples, at least one of the file server VMs shown in FIG. 2Amay have a different amount of memory allocated to it than other FSVMsshown in FIG. 2A. In some examples, a predetermined number of memoryallocation amounts may be used. For example, a ‘high memory capacity’and a ‘low memory capacity’ FSVM type may be defined, and the highmemory capacity FSVM type may have a higher amount of memory allocatedto it than the low memory capacity FSVM type.

The compute resource allocation for the FSVMs generally refers to theamount of compute resources allocated to the FSVM. In some examples, asame amount of compute resources may be allocated to each FSVM in avirtualized file server. In some examples, different amounts of computeresources may be allocated to different FSVMs in a virtualized fileserver. For example, each of the file server VMs shown in FIG. 2A mayhave a same amount of compute resources allocated to it. In otherexamples, at least one of the file server VMs shown in FIG. 2A may havea different amount of compute resources allocated to it than other FSVMsshown in FIG. 2A. In some examples, a predetermined number of computeresource allocation amounts may be used. For example, a ‘high computepower’ and a ‘low compute power’ FSVM type may be defined, and the highcompute power FSVM type may have a higher amount of compute resourcesallocated to it than the low compute power FSVM type.

In some examples, predetermined FSVM types may be defined based on acombination of memory and compute resource allocations. For example, a“primary compute” FSVM type may have a relatively high compute resourceallocation and a relatively low memory allocation. A “primary store”FSVM type may have a relatively high memory allocation and a relativelylow compute resource allocation.

Examples of FSVM managers described herein may re-allocate volume groupsin a storage pool of a virtualized file server responsive to a change inthe virtualized file server (e.g., a change in the number and/orconfiguration of FSVMs in the virtualized file server). The FSVMmanager, e.g., FSVM manager 804, may change the ownership of one or morevolume groups in a storage pool responsive to a change in FSVM numberand/or configuration. For example, during a scale-in of the virtualizedfile server, orphan volume groups formerly assigned to a FSVM that nolonger exists may be reassigned to one of the remaining FSVMs in thecluster. In another example, during a scale-out of the virtualized fileserver, volume groups may be moved to newly created FSVMs in thecluster. In either example, particular embodiments may further moveother volume groups around in order to more evenly distribute loadamongst the current set of FSVMs. In some examples, name serviceinformation and DNS information may also be updated for volume groupsthat are moved. In some examples, volume group reassignments may berolled back in the case of failure to add or remove FSVMs or failure tomove one or more volume groups.

Examples of systems described herein may include one or moreadministrator systems, such as admin system 808 of FIG. 8. Theadministrator system may be implemented using, for example, one or morecomputers, servers, laptops, desktops, tablets, mobile phones, or othercomputing systems. In some examples, the admin system 808 may be whollyand/or partially implemented using one of the computing nodes of thevirtualized environment. However, in some examples (such as shown inFIG. 8), the admin system 808 may be a different computing system fromthe virtualized system and may be in communication with a CVM of thevirtualized environment (e.g., virtualized environment 200 of FIG. 2A)using a wired or wireless connection (e.g., over a network).

Administrator systems described herein may host one or more userinterfaces, e.g., user interface 810. The user interface may beimplemented, for example, by displaying a user interface on a display ofthe administrator system. The user interface may receive input from oneor more users (e.g., administrators) using one or more input device(s)of the administrator system, such as, but not limited to, a keyboard,mouse, touchscreen, and/or voice input. The user interface 810 mayprovide the input to controller VM 110A (e.g., to the FSVM manager 804).The input may be used to provide a request to setup, install, and/orconfigure FSVMs described herein. The input may include one or moretarget number of connections and/or throughput as described herein. Theuser interface 810 may be implemented, for example, using a web serviceprovided by the controller VM 110A and/or FSVM manager 804 describedherein.

Examples of systems described herein may include one or more portalsystems, such as portal 812 of FIG. 8. The portal 812 may be implementedusing, for example, one or more computers, servers, laptops, desktops,tablets, mobile phones, or other computing systems. In some examples,the portal 812 may be wholly and/or partially implemented using one ofthe computing nodes of the virtualized environment (e.g., virtualizedenvironment 200 of FIG. 2A). However in some examples (such as shown inFIG. 8), the portal 812 may be a different computing system from thevirtualized system and may be in communication with a CVM of thevirtualized environment (e.g., virtualized environment 200 of FIG. 2A)using a wired or wireless connection (e.g., over a network).

Portal systems described herein may host one or more FSVM images, suchas FSVM image 814. The FSVM image 814 generally refers to software(e.g., executable instructions stored on one or more computer readablemedia) used to install one or more FSVMs (e.g. the FSVM 170 a, 170 b,and/or 170 c of FIG. 2A). In some examples, the FSVM image 814 may be ahypervisor-agnostic image. For example, the software stored as FSVMimage 814 may be in a format which is not specific to any particularhypervisor type. While FSVM image 814 is shown in FIG. 8, additionaland/or different images may be stored on portal 812 in some examples andused to install some or all of the system components shown in FIG. 2A,for example.

As an example of installations which may occur using systems describedherein, consider that, conventionally, NAS 128 comprises one or morededicated physical storage devices that must be physically attached tonetwork 140 and configured. In some examples, NAS 128 may be provided inthe form of one or more existing host machines 200 running CVMs 110.Examples of FSVM managers, such as FSVM manager 804, or anotherinstallation tool, may install and deploy VFS 202 (including FSVMs 170)to run as a service on top of NAS 128, using images obtained from portal812. In some examples, the controller VM 110A (e.g., using FSVM manager804) may identify a version of an existing FSVM running on a hostmachine. The FSVM manager 804 may upgrade the existing FSVM version ifthe FSVM image 814 is associated with a later version than the existingFSVM. Examples of FSVM managers may automatically upgrade FSVMs 170 uponreceiving a notification regarding availability of upgraded software forinstallation, e.g., on portal 812. In some examples, an installationtool (which may be running on one or more of the host systems of FIG.2A) may detect and resolve incompatibilities between one or more of thecontroller VM 110 a-c and VFS 202 (e.g., by upgrading the CVM software)before installing VFS 202, and then finally installing VFS 202 (e.g.,installing the FSVMs 107 a-c). In some examples, an administration toolmay provide a list of one or more downloadable versions of installersoftware for VFS 202 at a portal (e.g., portal 812) where the user mayselect one of the versions and commence the download.

In particular embodiments, the administration tool may handleinstallation, deployment, and upgrade of other elements of theinfrastructure depicted in FIGS. 2A-2B, such as, by way of example andnot limitation, CVMs 110, user VMs 101 and 102, or applications runningon the user VMs.

While shown as separate systems, in some examples, the admin system 808and/or portal 812 may be implemented using a same computing system toprovide both user interface 810 and FSVM image 814.

During operation, a controller virtual machine described herein (e.g.,controller VM 110A) may receive a request to change a distributed fileserver (e.g., virtualized file server 202). The controller VM 110A(e.g., using the FSVM manager 804) may change a number of FSVMs in thedistributed file server, an amount of memory allocated to the FSVMs inthe distributed file server, and/or an amount of compute resourcesallocated to the FSVMs responsive to the request. Moreover, thecontroller VM 110A may re-allocate one or more volume groups in astorage pool of the distributed file server responsive to the change.

The request may be received from a variety of sources. In some examples,an administrator or other user may request a change to the distributedfile server, e.g., using user interface 810. In some examples, thecontroller VM 110A, another controller VM in the virtualizedenvironment, and/or another process in the system may request thechange.

In some examples, the request to change the distributed file server maybe implemented using a request to scale out the distributed file server.To scale out the distributed file server, the controller VM 110A may addadditional FSVMs to the system. Any number of FSVMs may be added. Oncethe FSVMs are added, it may be desirable to distribute the file serverload across all FSVMs, including the added FSVMs. Accordingly, the CVM(e.g., using FSVM manager 804) may re-allocate volume groups such thatsome existing volume groups are owned by the newly added FSVMs. Forexample, ownership of certain volume groups previously owned by existingFSVMs in the distributed file server may be re-assigned to be owned bythe new FSVMs. An example of scaling out the distributed file serverwould be to change the virtualized file server 202 of FIG. 2A to have 5FSVMs instead of the three shown (e.g., adding two FSVMs).

In some examples, the request to change the distributed file server maybe implemented using a request to scale in the distributed file server.To scale in the distributed file server, the controller VM 110A mayremove a number of FSVMs from the system. For example, one or more FSVMsmay be terminated. Any number of FSVMs may generally be terminated inaccordance with a request to scale in. The CVM (e.g., using FSVM manager804) may re-allocate volume groups across the remaining FSVMs. Forexample, ownership of volume groups owned by the terminated FSVMs may bere-assigned to one or more of the remaining FSVMs.

In some examples, the request to change the distributed file server maybe implemented using a request to scale up the distributed file server.To scale up the distributed file server, the controller VM 110A (e.g.,using FSVM manager 804) may add additional memory, additional computeresources, or both to the FSVMs in the system. In some examples, all theFSVMs in the system may have added memory, compute resources, or both.In some examples, only certain ones of the FSVMs may have added memory,compute resources, or both responsive to the request to scale up. Anexample of scaling up the distributed file server would be to add morememory, compute resources, or both to one or more of the FSVMs 170 a-cin virtualized file server 202 of FIG. 2A. For example, the FSVM 170 amay be allocated 12 GB of memory and 4 CPUs. To scale up, the controllerVM 110A (e.g., using FSVM manager 804) may configure the FSVM 170 a tohave 24 GB of memory and 8 CPUs. Other amounts of memory and CPUs may beused in other examples.

In some examples, the request to change the distributed file server maybe implemented using a request to scale down the distributed fileserver. To scale down the distributed file server, the controller VM110A (e.g., using FSVM manager 804) may remove memory, computeresources, or both from one or more FSVMs in the system. In someexamples, some memory, compute resources, and/or both may be removedfrom all FSVMs in the system. In some examples, only certain ones of theFSVMs may have reduced memory, compute resource, or both responsive tothe request to scale down.

The changes described herein responsive to requests to scale up, scaledown, scale in and/or scale out the distributed file server may occurwhile the fileserver maintains accessible and servicing clients. Forexample, FSVM managers described herein may make changes describedherein without rendering the FSVM unavailable or unaccessible toclients.

In some examples, FSVM managers described herein (e.g., FSVM manager804) may manage installation of one or more FSVMs. For example, FSVMmanagers may install one or more FSVMs in a distributed system having noexisting distributed file system (e.g., an initial installation). Insome examples, FSVM managers may install one or more FSVMs responsive torequest to scale out and/or scale up an existing distributed fileserver.

Responsive to a request to install a distributed file server and/or oneor more FSVMs (e.g., a request to setup one or more FSVMs), thecontroller VM 110A (e.g., using FSVM manager 804) may image one or moreFSVMs. The FSVMs may be imaged by accessing a hypervisor-agnostic FSVMimage (e.g., FSVM image 814). The controller VM 110A (e.g., using FSVMmanager 804) may extract the hypervisor-agnostic FSVM image into ahypervisor-specific FSVM image for a particular hypervisor (e.g.,hypervisor 130 a, 130 b, and/or 130 c). For example, if the hypervisor130 a is a MICROSOFT HYPER-V hypervisor, the hypervisor-agnostic image(e.g., FSVM image 814) may be extracted to an image specific for aMICROSOFT HYPER-V hypervisor. Generally the file format the FSVM image814 may be transferred into a format specific to the MICROSOFT HYPER-Vhypervisor. The hypervisor-specific image may be used to install theFSVM on the particular host machine running the specific hypervisor.

For example, the FSVM image 814 may be implemented using ahypervisor-agnostic bootable image in a variety of hypervisorenvironments. The FSVM image 814 may be downloaded from portal 812 to ahypervisor storage container. The hypervisor-agnostic image may then beconverted into a native image that is specific to the hypervisorplatform (e.g., a hypervisor-specific image). Using the native image,FSVMs (or other VMs) may then be created and configured on thehypervisor, and network and storage configuration is completed. Finally,the VMs may be grouped into a cluster in order to provide a virtualizedservice in the form of an appliance. The hypervisor-agnostic bootableimage itself (e.g., FSVM image 814) may be compressed and prepended withheaders providing instructions regarding how to perform conversion ofthe hypervisor-agnostic bootable image (e.g., FSVM image 814) into anative image (e.g., a hypervisor-specific image).

The controller VM 110A (e.g., using FSVM manager 804) may make a numberof copies of the hyper-visor specific FSVM image and/or clone thehypervisor specific FSVM image a number of times when there are multipleFSVMs in the system having that specific hypervisor-type. For example,if both hypervisor 130 a and hypervisor 130 b are MICROSOFT HYPER-Vhypervisors, the FSVM manager 804 may extract the FSVM image 814 to ahypervisor-specific image for MICROSOFT HYPER-V, and may make a secondcopy of the hypervisor-specific image and/or clone thehypervisor-specific image. The first extracted file may be used to imageFSVM 170 a, and the copy and/or clone may be used to image the FSVM 170b. In this manner, the image may be cloned any number of times and theclones used to image a commensurate number of FSVMs. In other examples,generally any number of copies may be made and used to image acommensurate number of FSVMs.

In some examples, a request to install a distributed file server and/orto change one or more FSVMs in a distributed file server may include atarget number of connections and a target throughput of the distributedfile server. For example, the user interface 810 may prompt a requestorfor these target values. As described herein, controller VM 110A maydetermine a number of FSVMs, an amount of memory allocated to the FSVMs,an amount of compute resources allocated to the FSVMs, or combinationsthereof, based on the target number of connections and the targetthroughput. The determination may be made with reference to FSVM configdata 806 which may include stored associations between the target numberof connections and the target throughput and the number of FSVMs, theamount of memory, and the amount of compute resources. The controller VM110A (e.g., using the FSVM manager 804) may setup the number of FSVMs asdetermined, and may configure the FSVMs to have the determined amount ofmemory and amount of compute resources. For example, the controller VM110A (e.g., using the FSVM manager 804) may communicate with ahypervisor associated with a particular FSVM to configure the FSVM tohave the desired amount of memory and amount of compute resources.

FIG. 9 is an illustration of a view of a user interface arranged inaccordance with examples described herein. The view 900 may be a view ofuser interface 810 of FIG. 8 at a particular time. For example, the view900 may be displayed on a display of admin system 808. The view 900includes input area 902 and input area 904. Additional, fewer, and/ordifferent areas or input prompts may be provided in other examples.

The view 900 includes a prompt (e.g., “concurrent connections”) for auser to enter a target number of connections into input area 902. Theuser may enter the target number of connections using an input device ofthe admin system 808 (e.g., a keyboard). In the example of FIG. 9, theview 900 is shown with a user having entered the target number ofconcurrent connections as 750, Any number of concurrent connections maybe requested. Generally, the number of connections may be correlatedwith a number of users of the virtualized file server being installedand/or changed by the user interface. For example, the number ofconnections may refer to the number of intended users of a virtualizedfile server.

The view 900 includes another prompt (e.g., “throughput in MBPS”) for auser to enter a target throughput for the virtualized file server ininput area 904. The user may enter the target throughput using an inputdevice of the admin system 808 (e.g., a keyboard). In the example ofFIG. 9, the view 900 is shown with a user having entered the targetthroughput as 270 Mbps. Other throughputs and/or units may be used inother examples. Generally, the target throughput refers to the amount ofdata per unit time that may be provided from the virtualized fileserver.

After entering the target number of connections in input area 902 andthe target throughput in input area 904, the user may provide anindication (e.g., by clicking the ‘save’ button shown in FIG. 9) toproceed with a request to install and/or update a virtualized fileserver. The user interface 810 may accordingly include the number ofconnections and the throughput received in input area 902 and input area904 in a request which may be provided to one or more FSVM managersdescribed herein, such as FSVM manager 804.

FIG. 10 is an illustration of a view of a user interface arranged inaccordance with examples described herein. The view 1000 may be a viewof user interface 810 of FIG. 8 at a particular time. For example, theview 1000 may be displayed on a display of admin system 808. The view1000 includes user input parameters 1002 and recommended configuration1004. Additional, fewer, and/or different areas, input prompts, ordisplayed information may be provided in other examples.

The view 1000 may in some examples be displayed following user input ofthe target number of connections and target throughput (e.g., as shownin FIG. 9). As described herein, FSVM managers, e.g., FSVM manager 804of FIG. 8 may determine a number of FSVMs for user in a virtualized fileserver, and a configuration of the FSVMs (e.g., a memory and/orcomputing resource allocation for the FSVMs) based on an input targetnumber of connections and target throughput. In some examples, prior toinstallation, the recommended number of FSVMs and configuration of theFSVMs may be displayed to the requestor (e.g., as shown in view 1000).

The view 1000 includes user input parameters 1002. For example, thetarget 750 connections and 270 MBPS throughput input as shown in FIG. 9may be displayed. The view 1000 includes recommended configuration 1004.The recommended configuration 1004 may include a display of the numberof FSVMs (3 in FIG. 10), the recommended compute resources allocated perFSVM (4 virtual CPUS, VCPUs, as shown in FIG. 10), and the recommendedmemory allocation per FSVM (12 GB as shown in FIG. 10) as determined byan FSVM manager responsive to the user input parameters 1002. Othernumbers may be used in other examples. While the recommendedconfiguration 1004 illustrates one amount of compute resources andmemory to be allocated to each of the FSVMs, in other examples, therecommended configuration 1004 may display different amounts of computeresources and/or memory to be allocated for each of the recommendedFSVMs.

In some examples, a user may revise the recommendations generated byFSVM manager and displayed in recommended configuration 1004 prior toinitiating an installation and/or change of a virtualized file server.For example, the recommended configuration 1004 area may be editable bythe user prior to initiating a request (e.g., clicking the “Next” buttonin view 1000). On initiation of the request, the FSVM manager may setupthe requested number of FSVMs with the memory and compute resourceallocations shown in recommended configuration 1004.

The view 1000 additionally displays the name (e.g., “afs-fileserver”) ofthe requested virtualized file server, the domain (e.g., “nutanix.com”)of the requested virtualized file server, and the total storage (e.g., 1TiB) of the requested virtualized file server.

FIG. 3A illustrates an example hierarchical structure 300 of a VFSinstance in a cluster according to particular embodiments. A Cluster 302contains two VFS instances, FS1 304 and FS2 306. Each VFS instance maybe identified by a name such as “\\instance”, e.g., “\\FS1” for WINDOWSfilesystems, or a name such as “instance”, e.g., “FS1” for UNIX-typefilesystems. The VFS instance FS1 304 contains shares, including Share-1308 and Share-2 310. Shares may have names such as “Users” for a sharethat stores user home directories, or the like. Each share may have apath name such as \\FS1\Share-1 or \\FS1\Users. As an example and not byway of limitation, a share may correspond to a disk partition or a poolof filesystem blocks on WINDOWS and UNIX-type filesystems. As anotherexample and not by way of limitation, a share may correspond to a folderor directory on a VFS instance 304. Shares may appear in the filesysteminstance 202 as folders or directories to users of user VMs 101 a.Share-1 308 includes two folders, Folder-1 312, and Folder-2 314, andmay also include one or more files (e.g., files not in folders). Eachfolder 312, 314 may include one or more files 318. Share-2 310 includesa folder Folder-3 316, which includes a file File-2 320. Each folder hasa folder name such as “Folder-1”, “Users”, or “Sam” and a path name suchas “\\FS1\Share-1\Folder-1” (WINDOWS) or “share-1:/fs1/Users/Sam”(UNIX). Similarly, each file has a file name such as “File-1” or“Forecast.xls” and a path name such as “\\FS1\Share-1\Folder-1 \File-1”or “share-1:/fs1/Users/Sam/Forecast.xls”.

FIG. 3B illustrates two example host machines 200 a and 200 b, eachproviding file storage services for portions of two VFS instances FS1and FS2 according to particular embodiments. The first host machine,Host-1 200 a, includes two user VMs 101 a, 102 a, a Hypervisor 130 a, aFSVM named FileServer-VM-1 (abbreviated FSVM-1) 170 a, aController/Service VM named CVM-1 110 a, and local storage 122 a.Host-1's FileServer-VM-1 170 a has an IP (Internet Protocol) networkaddress of 10.1.1.1, which is an address of a network interface onHost-1 200 a. Host-1 has a hostname ip-addr1, which may correspond toHost-1's IP address 10.1.1.1. The second host machine, Host-2 200 b,includes two user VMs 101 b, 102 b, a Hypervisor 130 b, a File Server VMnamed FileServer-VM-2 (abbreviated FSVM-2) 170 b, a Controller/ServiceVM named CVM-2 110 b, and local storage 122 b. Host-2's FileServer-VM-1170 b has an IP network address of 10.1.1.2, which is an address of anetwork interface on Host-2 200 b.

In particular embodiments, filesystems FileSystem-1A 364 a andFileSystem-2A 365 a implement the structure of files and folders forportions of the FS1 and FS2 file server instances, respectively, thatare located on (e.g., served by) FileServer-VM-1 170 a on Host-1 200 a.Other file systems on other host machines may implement other portionsof the FS1 and FS2 file server instances. The filesystems 364 a and 365a may implement the structure of at least a portion of a file serverinstance by translating file system operations, such as opening a file,writing data to or reading data from the file, deleting a file, and soon, to disk I/O operations such as seeking to a portion of the disk,reading or writing an index of file information, writing data to orreading data from blocks of the disk, allocating or de-allocating theblocks, and so on. The filesystems 364 a, 365 a may thus store theirfilesystem data, including the structure of the folder and filehierarchy, the names of the storage items (e.g., folders and files), andthe contents of the storage items on one or more storage devices, suchas local storage 122 a. The particular storage device or devices onwhich the filesystem data for each filesystem are stored may bespecified by an associated filesystem pool (e.g., 366 a-c and 367 a-c).For example, the storage device(s) on which data for FileSystem-1A 364 aand FileSystem-2A, 365 a are stored may be specified by respectivefilesystem pools FS1-Pool-1 366 a and FS2-Pool-2 367 a. The storagedevices for the pool 366 a may be selected from volume groups providedby CVM-1 110 a, such as volume group VG1 368 a and volume group VG2 369a. Each volume group 368 a, 369 a may include a group of one or moreavailable storage devices that are present in local storage 122 aassociated with (e.g., by iSCSI communication) the CVM-1 110 a. TheCVM-1 110 a may be associated with a local storage 122 a on the samehost machine 200 a as the CVM-1 110 a, or with a local storage 122 b ona different host machine 200 b. The CVM-1 110 a may also be associatedwith other types of storage, such as cloud storage 126, networkedstorage 128 or the like. Although the examples described herein includeparticular host machines, virtual machines, file servers, file serverinstances, file server pools, CVMs, volume groups, and associationstherebetween, any number of host machines, virtual machines, fileservers, file server instances, file server pools, CVMs, volume groups,and any associations therebetween are possible and contemplated.

In particular embodiments, the filesystem pool 366 a may associate anystorage device in one of the volume groups 368 a, 369 a of storagedevices that are available in local storage 122 a with the filesystemFileSystem-1A 364 a. For example, the filesystem pool FS1-Pool-1 366 amay specify that a disk device named hd1 in the volume group VG1 368 aof local storage 122 a is a storage device for FileSystem-1A 364 a forfile server FS1 on FSVM-1 170 a. A filesystem pool FS2-Pool-2 367 a mayspecify a storage device FileSystem-2A 365 a for file server FS2 onFSVM-1 170 a. The storage device for FileSystem-2A 365 a may be, e.g.,the disk device hd1, or a different device in one of the volume groups368 a, 369 a, such as a disk device named hd2 in volume group VG2 369 a.Each of the filesystems FileSystem-1A 364 a, FileSystem-2A 365 a may be,e.g., an instance of the NTFS file system used by the WINDOWS operatingsystem, of the UFS Unix file system, or the like. The term “filesystem”may also be used herein to refer to an instance of a type of filesystem,e.g., a particular structure of folders and files with particular namesand content.

In one example, referring to FIG. 3A, an FS1 hierarchy rooted at FileServer FS1 304 may be located on FileServer-VM-1 170 a and stored infilesystem instance FileSystem-1A 364 a. That is, the filesysteminstance FileSystem-1A 364 a may store the names of the shares andstorage items (such as folders and files), as well as the contents ofthe storage items, shown in the hierarchy at and below File Server FS1304. A portion of the FS1 hierarchy shown in FIG. 3A, such the portionrooted at Folder-2 314, may be located on FileServer-VM-2-170 b onHost-2 200 b instead of FileServer-VM-1-170 a, in which case thefilesystem instance FileSystem-1B 364 b may store the portion of the FS1hierarchy rooted at Folder-2 314, including Folder-3 314, Folder-4 322and File-3 324. Similarly, an FS2 hierarchy rooted at File Server FS2306 in FIG. 3A may be located on FileServer-VM-1 170 a and stored infilesystem instance FileSystem-2A 365 a. The FS2 hierarchy may be splitinto multiple portions (not shown), such that one portion is located onFileServer-VM-1 170 a on Host-1 200 a, and another portion is located onFileServer-VM-2 170 b on Host-2 200 b and stored in filesystem instanceFileSystem-2B 365 c.

In particular embodiments, FileServer-VM-1 (abbreviated FSVM-1) 170 a onHost-1 200 a is a leader for a portion of file server instance FS1 and aportion of FS2, and is a backup for another portion of FS1 and anotherportion of FS2. The portion of FS1 for which FileServer-VM-1 170 a is aleader corresponds to a storage pool labeled FS1-Pool-1 366 a.FileServer-VM-1 is also a leader for FS2-Pool-2 367 a, and is a backup(e.g., is prepared to become a leader upon request, such as in responseto a failure of another FSVM) for FS1-Pool-3 366 b and FS2-Pool-4 367 bon Host-2. In particular embodiments, FileServer-VM-2 (abbreviatedFSVM-2) 170 b is a leader for a portion of file server instance FS1 anda portion of FS2, and is a backup for another portion of FS1 and anotherportion of FS2. The portion of FS1 for which FSVM-2 170 b is a leadercorresponds to a storage pool labeled FS1-Pool-3 366 b. FSVM-2 170 b isalso a leader for FS2-Pool-4 367 b, and is a backup for FS1-Pool-1 366 aand FS2-Pool-2 367 a on Host-1.

In particular embodiments, the file server instances FS1, FS2 providedby the FSVMs 170 a and 170 b may be accessed by user VMs 101 a and 101 bvia a network filesystem protocol such as SMB, CIFS, NFS, or the like.Each FSVM 170 a and 170 b may provide what appears to clientapplications on user VMs 101 a and 101 b to be a single filesysteminstance, e.g., a single namespace of shares, files and folders, foreach file server instance 202. However, the shares, files, and foldersin a file server instance such as FS1 may actually be distributed acrossmultiple FSVMs 170 a and 170 b. For example, different folders in thesame file server instance may be associated with different correspondingFSVMs 170 a and 170 b and CVMs 110 a and 110 b on different hostmachines 200 a and 200 b.

The example file server instance FS1 304 shown in FIG. 3A has twoshares, Share-1 308 and Share-2 310. Share-1 308 may be located onFSVM-1 170 a, CVM-1 110 a, and local storage 122 a. Network file systemprotocol requests from user VMs 101 and 102 to read or write data onfile server instance FS1 304 and any share, folder, or file in theinstance may be sent to FSVM-1 170 a. FSVM-1 170 a may determine whetherthe requested data, e.g., the share, folder, file, or a portion thereof,referenced in the request, is located on FSVM-1, and FSVM-1 is a leaderfor the requested data. If not, FSVM-1 may respond to the requestingUser-VM with an indication that the requested data is not covered by(e.g., is not located on or served by) FSVM-1. Otherwise, the requesteddata is covered by (e.g., is located on or served by) FSVM-1, so FSVM-1may send iSCSI protocol requests to a CVM that is associated with therequested data. Note that the CVM associated with the requested data maybe the CVM-1 110 a on the same host machine 200 a as the FSVM-1, or adifferent CVM on a different host machine 200 b, depending on theconfiguration of the VFS 202. In this example, the requested Share-1 islocated on FSVM-1, so FSVM-1 processes the request. To provide for pathavailability, multipath I/O (MPIO) may be used for communication withthe FSVM, e.g., for communication between FSVM-1 and CVM-1. The activepath may be set to the CVM that is local to the FSVM (e.g., on the samehost machine) by default. The active path may be set to a remote CVMinstead of the local CVM, e.g., when a failover occurs.

Continuing with the data request example, the associated CVM is CVM 110a, which may in turn access the storage device associated with therequested data as specified in the request, e.g., to write specifieddata to the storage device or read requested data from a specifiedlocation on the storage device. In this example, the associated storagedevice is in local storage 122 a, and may be an HDD or SSD. CVM-1 110 amay access the HDD or SSD via an appropriate protocol, e.g., iSCSI,SCSI, SATA, or the like. CVM 110 a may send the results of accessinglocal storage 122 a, e.g., data that has been read, or the status of adata write operation, to CVM 110 a via, e.g., SATA, which may in turnsend the results to FSVM-1 170 a via, e.g., iSCSI. FSVM-1 170 a may thensend the results to user VM 101 a via SMB through the Hypervisor 130 a.

Share-2 310 may be located on FSVM-2 170 b, on Host-2. Network fileservice protocol requests from user VMs 101 a and 101 b to read or writedata on Share-2 may be directed to FSVM-2 170 b on Host-2 by other FSVMs170 a. Alternatively, user VMs 101 a and 101 b may send such requestsdirectly to FSVM-2 170 b on Host-2, which may process the requests usingCVM-2 110 b and local storage 122 b on Host-2 as described above forFSVM-1 170 a on Host-1.

A file server instance 202 such as FS1 304 in FIG. 3A may appear as asingle filesystem instance (e.g., a single namespace of folders andfiles that are accessible by their names or pathnames without regard fortheir physical locations), even though portions of the filesystem arestored on different host machines 200 a-c. Since each FSVM 170 mayprovide a portion of a file server instance 202, each FSVM 170 may haveone or more “local” filesystems 364 a, 365 a that provide the portion ofthe file server instance 202 (e.g., the portion of the namespace offiles and folders) associated with the FSVM 170.

FIG. 3C illustrates example interactions between a client 330 and hostmachines 200 a and 200 c on which different portions of a VFS instanceare stored according to particular embodiments. A client 330, e.g., anapplication program executing in one of the user VMs 101 and 102 on thehost machines 200 a-c of FIGS. 2A-2B (e.g. user VM 101 b on host machine200 b) requests access to a folder \\FS1.domain.name\Share-1\Folder-3.The request may be in response to an attempt to map\\FS1.domain.name\Share-1 to a network drive in the operating systemexecuting in the user VM 101 c followed by an attempt to access thecontents of Share-1 or to access the contents of Folder-3, such aslisting the files in Folder-3.

FIG. 3C shows interactions that occur between the client 330, FSVMs 170a and 170 b on host machines 200 a and 200 b, and a name server 332 whena storage item is mapped or otherwise accessed. The name server 332 maybe provided by a server computer system, such as one or more of the hostmachines 200, or a server computer system separate from the hostmachines 200. In one example, the name server 332 may be provided by anACTIVE DIRECTORY service executing on one or more computer systems andaccessible via the network 140. The interactions are shown as arrowsthat represent communications, e.g., messages sent via the network 140.Note that the client 330 may be executing in a user VM 101, which may beco-located with one of the FSVMs 170 a and 170 b. In such a co-locatedcase, the arrows between the client 330 and the host machine 200 onwhich the FSVM 170 is located may represent communication within thehost machine 200, and such intra-host machine communication may beperformed using a mechanism different from communication over thenetwork 140, e.g., shared memory or inter process communication.

In particular embodiments, when the client 330 requests access toFolder-3, a VFS client component executing in the user VM 101 b may usea distributed file system protocol such as MICROSOFT DFS, or the like,to send the storage access request to one or more of the FSVMs 170 a-cof FIGS. 2A-2B. To access the requested file or folder, the clientdetermines the location of the requested file or folder, e.g., theidentity and/or network address of the FSVM 170 on which the file orfolder is located. The client may query a domain cache of FSVM 170 a-cnetwork addresses that the client has previously identified (e.g.,looked up). If the domain cache contains the network address of an FSVM170 associated with the requested folder name\\FS1.domain.name\Share-1\Folder-3, then the client retrieves theassociated network address from the domain cache and sends the accessrequest to the network address, starting at step 393 as described below.

In particular embodiments, at step 381, the client may send a requestfor a list of addresses of FSVMs 170 a-170 c to a name server 332. Thename server 332 may be, e.g., a DNS server or other type of server, suchas a MICROSOFT domain controller (not shown), that has a database ofFSVM addresses. At step 382, the name server 332 may send a reply thatcontains a list of FSVM 170 network addresses, e.g., ip-addr1, ip-addr2,and ip-addr3, which correspond to the FSVMs 170 a-c in this example. Atstep 383, the client 330 may send an access request to one of thenetwork addresses, e.g., the first network address in the list (ip-addr1in this example), requesting the contents of Folder-3 of Share-1. Byselecting the first network address in the list, the particular FSVM 170to which the access request is sent may be varied, e.g., in around-robin manner by enabling round-robin DNS (or the like) on the nameserver 332. The access request may be, e.g., an SMB connect request, anNFS open request, and/or appropriate request(s) to traverse thehierarchy of Share-1 to reach the desired folder or file, e.g., Folder-3in this example.

At step 384, FileServer-VM-1 170 a may process the request received atstep 383 by searching a mapping or lookup table, such as a sharding map360 a, for the desired folder or file. The map 360 maps stored objects,such as shares, folders, or files, to their corresponding locations,e.g., the names or addresses of FSVMs 170. The map 360 may have the samecontents on each host machine 200, with the contents on different hostmachines being synchronized using a distributed data store as describedbelow. For example, the map 360 a may contain entries that map Share-1and Folder-1 to the File Server FSVM-1 170 a, and Folder-3 to the FileServer FSVM-3 170 c. An example map 360 is shown in Table 1 below.

TABLE 1 Stored Object Location Folder-1 FSVM-1 Folder-2 FSVM-1 File-1FSVM-1 Folder-3 FSVM-3 File-2 FSVM-3

In particular embodiments, the map 360 may be accessible on each of thehost machines 200. As described with reference to FIGS. 2A-2B, the maps360 a and 360 c may be copies of a distributed data structure that aremaintained and accessed at each FSVM 170 a-c using a distributed dataaccess coordinator 370 a and 370 c. The distributed data accesscoordinator 370 a and 370 c may be implemented based on distributedlocks or other storage item access operations. Alternatively, thedistributed data access coordinator 370 a and 370 c may be implementedby maintaining a master copy of the maps 360 a and 360 c at a leadernode such as the host machine 200 c, and using distributed locks toaccess the master copy from each FSVM 170 a and 170 b. The distributeddata access coordinator 370 a and 370 c may be implemented usingdistributed locking, leader election, or related features provided by acentralized coordination service for maintaining configurationinformation, naming, providing distributed synchronization, and/orproviding group services (e.g., APACHE ZOOKEEPER or other distributedcoordination software). Since the map 360 a indicates that Folder-3 islocated at FSVM-3 170 c on Host-3 200 c, the lookup operation at step384 determines that Folder-3 is not located at FSVM-1 on Host-1 200 a.Thus, at step 385 the FSVM-1 170 a sends a response, e.g., a “NotCovered” DFS response, to the client 330 indicating that the requestedfolder is not located at FSVM-1. At step 386, the client 330 sends arequest to FSVM-1 for a referral to the FSVM on which Folder-3 islocated. FSVM-1 uses the map 360 a to determine that Folder-3 is locatedat FSVM-3 on Host-3 200 c, and at step 387 returns a response, e.g., a“Redirect” DFS response, redirecting the client 330 to FSVM-3. Theclient 330 may then determine the network address for FSVM-3, which isip-addr3 (e.g., a host name “ip-addr3.domain.name” or an IP address,10.1.1.3). The client 330 may determine the network address for FSVM-3by searching a cache stored in memory of the client 330, which maycontain a mapping from FSVM-3 to ip-addr3 cached in a previousoperation. If the cache does not contain a network address for FSVM-3,then at step 388 the client 330 may send a request to the name server332 to resolve the name FSVM-3. The name server may respond with theresolved address, ip-addr3, at step 389. The client 330 may then storethe association between FSVM-3 and ip-addr3 in the client's cache.

In particular embodiments, failure of FSVMs 170 may be detected usingthe centralized coordination service. For example, using the centralizedcoordination service, each FSVM 170 a may create a lock on the hostmachine 200 a on which the FSVM 170 a is located using ephemeral nodesof the centralized coordination service (which are different from hostmachines 200 but may correspond to host machines 200). Other FSVMs 170 band 170 c may volunteer for leadership of resources of remote FSVMs 170on other host machines 200, e.g., by requesting a lock on the other hostmachines 200. The locks requested by the other nodes are not grantedunless communication to the leader host machine 200 c is lost, in whichcase the centralized coordination service deletes the ephemeral node andgrants the lock to one of the volunteer host machines 200 a and 200 b,which becomes the new leader. For example, the volunteer host machines200 a and 200 b may be ordered by the time at which the centralizedcoordination service received their requests, and the lock may begranted to the first host machine 200 on the ordered list. The firsthost machine 200 (e.g., host machine 200 b) on the list may thus beselected as the new leader. The FSVM 170 b on the new leader hasownership of the resources that were associated with the failed leaderFSVM 170 a until the failed leader FSVM 170 c is restored, at whichpoint the restored FSVM 170 a may reclaim the local resources of thehost machine 200 c on which it is located.

At step 390, the client 330 may send an access request to FSVM-3 170 cat ip-addr3 on Host-3 200 c requesting the contents of Folder-3 ofShare-1. At step 391, FSVM-3 170 c queries FSVM-3's copy of the map 360using FSVM-3's instance of the distributed data access coordinator 370c. The map 360 indicates that Folder-3 is located on FSVM-3, so at step392 FSVM-3 accesses the file system 364 c to retrieve information aboutFolder-3 316 and its contents (e.g., a list of files in the folder,which includes File-2 320) that are stored on the local storage 122 c.FSVM-3 may access local storage 122 c via CVM-3 110 c, which providesaccess to local storage 122 c via a volume group 368 c that contains oneor more volumes stored on one or more storage devices in local storage122 c. At step 393, FSVM-3 may then send the information about Folder-3and its contents to the client 330. Optionally, FSVM-3 may retrieve thecontents of File-2 and send them to the client 330, or the client 330may send a subsequent request to retrieve File-2 as needed.

FIG. 3D illustrates an example virtualized file server having a failovercapability according to particular embodiments. To provide highavailability, e.g., so that the file server continues to operate afterfailure of components such as a CVM, FSVM, or both, as may occur if ahost machine fails, components on other host machines may take over thefunctions of failed components. When a CVM fails, a CVM on another hostmachine may take over input/output operations for the failed CVM.Further, when an FSVM fails, an FSVM on another host machine may takeover the network address and CVM or volume group that were being used bythe failed FSVM. If both an FSVM and an associated CVM on a host machinefail, as may occur when the host machine fails, then the FSVM and CVM onanother host machine may take over for the failed FSVM and CVM. When thefailed FSVM and/or CVM are restored and operational, the restored FSVMand/or CVM may take over the operations that were being performed by theother FSVM and/or CVM. In FIG. 3D, FSVM-1 170 a communicates with CVM-1110 a to use the data storage in volume groups VG1 368 a and VG2 369 a.For example, FSVM-1 is using disks in VG1 and VG2, which are iSCSItargets. FSVM-1 has iSCSI initiators that communicate with the VG1 andVG2 targets using MPIO (e.g., DM-MPIO on the LINUX operating system).FSVM-1 may access the volume groups VG1 and VG2 via in-guest iSCSI.Thus, any FSVM may connect to any iSCSI target if an FSVM failureoccurs.

In particular embodiments, during failure-free operation, there areactive iSCSI paths between FSVM-1 and CVM-1, as shown in FIG. 3D by thedashed lines from the FSVM-1 filesystems for FS1 364 a and FS2 365 a toCVM-1's volume group VG1 368 a and VG2 369 a, respectively. Further,during failure-free operation there are inactive failover (e.g.,standby) paths between FSVM-1 and CVM-3 110 c, which is located onHost-3. The failover paths may be, e.g., paths that are ready to beactivated in response to the local CVM CVM-1 becoming unavailable. Theremay be additional failover paths that are not shown in FIG. 3D. Forexample, there may be failover paths between FSVM-1 and a CVM on anotherhost machine, such as CVM-2 110 b on Host-2 200 b. The local CVM CVM-1110 a may become unavailable if, for example, CVM-1 crashes, or the hostmachine on which the CVM-1 is located crashes, loses power, losesnetwork communication between FSVM-1 170 a and CVM-1 110 a. As anexample and not by way of limitation, the failover paths do not performI/O operations during failure-free operation. Optionally, metadataassociated with a failed CVM 110 a, e.g., metadata related to volumegroups 368 a, 369 a associated with the failed CVM 110 a, may betransferred to an operational CVM, e.g., CVM 110 c, so that the specificconfiguration and/or state of the failed CVM 110 a may be re-created onthe operational CVM 110 c.

FIG. 3E illustrates an example virtualized file server that hasrecovered from a failure of Controller/Service VM CVM-1 110 a byswitching to an alternate Controller/Service VM CVM-3 110 c according toparticular embodiments. When CVM-1 110 a fails or otherwise becomesunavailable, then the FSVM associated with CVM-1, FSVM-1 170 a, maydetect a PATH DOWN status on one or both of the iSCSI targets for thevolume groups VG1 368 a and VG2 369 a, and initiate failover to a remoteCVM that can provide access to those volume groups VG1 and VG2. Forexample, when CVM-1 110 a fails, the iSCSI MPIO may activate failover(e.g., standby) paths to the remote iSCSI target volume group(s)associated with the remote CVM-3 110 c on Host-3 200 c. CVM-3 providesaccess to volume groups VG1 and VG2 as VG1 368 c and VG2 369 c, whichare on storage device(s) of local storage 122 c. The activated failoverpath may take over I/O operations from failed CVM-1 110 a. Optionally,metadata associated with the failed CVM-1 110 a, e.g., metadata relatedto volume groups 368 a, 369 a, may be transferred to CVM-3 so that thespecific configuration and/or state of CVM-1 may be re-created on CVM-3.When the failed CVM-1 again becomes available, e.g., after it has beenre-started and has resumed operation, the path between FSVM-1 and CVM-1may reactivated or marked as the active path, so that local I/O betweenCVM-1 and FSVM-1 may resume, and the path between CVM-3 and FSVM-1 mayagain become a failover (e.g., standby) path.

FIG. 3F illustrates an example virtualized file server that hasrecovered from failure of a FSVM by electing a new leader FSVM accordingto particular embodiments. When an FSVM-2 170 b fails, e.g., because ithas been brought down for maintenance, has crashed, the host machine onwhich it was executing has been powered off or crashed, networkcommunication between the FSVM and other FSVMs has become inoperative,or other causes, then the CVM that was being used by the failed FSVM,the CVM's associated volume group(s), and the network address of thehost machine on which the failed FSVM was executing may be taken over byanother FSVM to provide continued availability of the file services thatwere being provided by the failed FSVM. In the example shown in FIG. 3F,FSVM-2 170 b on Host-2 200 b has failed. One or more other FSVMs, e.g.,FSVM-1 170 a or FSVM-3 170 c, or other components located on one or moreother host machines, may detect the failure of FSVM-2, e.g., bydetecting a communication timeout or lack of response to a periodicstatus check message. When FSVM-2's failure is detected, an election maybe held, e.g., using a distributed leader election process such as thatprovided by the centralized coordination service. The host machine thatwins the election may become the new leader for the filesystem pools 366b, 367 b for which the failed FSVM-2 was the leader. In this example,FSVM-1 170 a wins the election and becomes the new leader for the pools366 b, 367 b. FSVM-1 170 a thus attaches to CVM-2 110 b by creating filesystem 364 b, 365 c instances for the file server instances FS1 and FS2using FS1-Pool-3 366 b and FS2-Pool-4 367 b, respectively. In this way,FSVM-1 takes over the filesystems and pools for CVM-2's volume groups,e.g., volume groups VG1 366 b and VG2 367 b of local storage 122 b.Further, FSVM-1 takes over the IP address associated with FSVM-2,10.1.1.2, so that storage access requests sent to FSVM-2 are receivedand processed by FSVM-1. Optionally, metadata used by FSVM-1, e.g.,metadata associated with the filesystems, may be transferred to FSVM-3so that the specific configuration and/or state of the filesystems maybe re-created on FSVM-3. Host-2 200 b may continue to operate, in whichcase CVM-2 110 b may continue to execute on Host-2. When FSVM-2 againbecomes available, e.g., after it has been re-started and has resumedoperation, FSVM-2 may assert leadership and take back its IP address(10.1.1.2) and storage (FS1-Pool-3 366 b and FS2-Pool-4 367 b) fromFSVM-1.

FIGS. 3G and 3H illustrate example virtualized file servers that haverecovered from failure of a host machine 200 a by switching to anotherController/Service VM and another FSVM according to particularembodiments. The other Controller/Service VM and FSVM are located on asingle host machine 200 c in FIG. 3G, and on two different host machines200 b, 200 c in FIG. 3H. In both FIGS. 3G and 3H, Host-1 200 a hasfailed, e.g., crashed or otherwise become inoperative or unresponsive tonetwork communication. Both FSVM-1 170 a and CVM-1 110 a located on thefailed Host-1 200 a have thus failed. Note that the CVM 110 a and FSVM170 a on a particular host machine 200 a may both fail even if the hostmachine 200 a itself does not fail. Recovery from failure of a CVM 110 aand an FSVM 170 a located on the same host machine 200 a, regardless ofwhether the host machine 200 a itself failed, may be performed asfollows. The failure of FSVM-1 and CVM-1 may be detected by one or moreother FSVMs, e.g., FSVM-2 170 b, FSVM-3 170 c, or by other componentslocated on one or more other host machines. FSVM-1's failure may bedetected when a communication timeout occurs or there is no response toa periodic status check message within a timeout period, for example.CVM-1's failure may be detected when a PATH DOWN condition occurs on oneor more of CVM-1's volume groups' targets (e.g., iSCSI targets).

When FSVM-1's failure is detected, an election may be held as describedabove with reference to FIG. 3F to elect an active FSVM to take overleadership of the portions of the file server instance for which thefailed FSVM was the leader. These portions are FileSystem-1A 364 a forthe portion of file server FS1 located on FSVM-1, and FileSystem-2A 365a for the portion of file server FS2 located on FSVM-1. FileSystem-1A364 a uses the pool FS-Pool-1 366 a and FileSystem-2A 365 a uses thepool FS2-Pool-2 367 a. Thus, the FileSystem-1A 364 a and FileSystem-2Amay be re-created on the new leader FSVM-3 170 c on Host-3 200 c.Further, FSVM-3 170 c may take over the IP address associated withfailed FSVM-1 170 a, 10.1.1.1, so that storage access requests sent toFSVM-1 are received and processed by FSVM-3.

One or more failover paths from an FSVM to volume groups on one or moreCVMs may be defined for use when a CVM fails. When CVM-1's failure isdetected, the MPIO may activate one of the failover (e.g., standby)paths to remote iSCSI target volume group(s) associated with a remoteCVM. For example, there may be a first predefined failover path fromFSVM-1 to the volume groups VG1 368 c, 369 c in CVM-3 (which are on thesame host as FSVM-1 when FSVM-1 is restored on Host-3 in examples ofFIGS. 3G and 3H), and a second predefined failover path to the volumegroups VG1 368 b, VG2 369 b in CVM-2. The first failover path, to CVM-3,is shown in FIG. 3G, and the second failover path, to CVM-2 is shown inFIG. 3H. An FSVM or MPIO may choose the first or second failover pathaccording to the predetermined MPIO failover configuration that has beenspecified by a system administrator or user. The failover configurationmay indicate that the path is selected (a) by reverting to the previousprimary path, (b) in order of most preferred path, (c) in a round-robinorder, (d) to the path with the least number of outstanding requests,(e) to the path with the least weight, or (f) to the path with the leastnumber of pending requests. When failure of CVM-1 110 a is detected,e.g., by FSVM-1 or MPIO detecting a PATH DOWN condition on one ofCVM-1's volume groups VG1 368 a or VG2 369 a, the alternate CVM on theselected failover path may take over I/O operations from the failedCVM-1. As shown in FIG. 3G, if the first failover path is chosen, CVM-3110 c on Host-3 200 c is the alternate CVM, and the pools FS1-Pool-1 366a and FS2-Pool-2 367 a, used by the filesystems FileSystem-1A 364 a andFileSystem-2A 365 a, respectively, which have been restored on FSVM-3 onHost-3, may use volume groups VG1 368 c and VG2 369 c of CVM-3 110 c onHost-3 when the first failover path is chosen. Alternatively, as shownin FIG. 3H, if the second failover path is chosen, CVM-2 on Host-2 isthe alternate CVM, and the pools FS1-Pool-1 366 a and FS2-Pool-2 367 aused by the respective filesystems FileSystem-1A 364 a and FileSystem-2A365 a, which have been restored on FSVM-3, may use volume groups VG1 368b and VG2 369 b on Host-2, respectively.

Optionally, metadata used by FSVM-1 170 a, e.g., metadata associatedwith the filesystems, may be transferred to FSVM-3 as part of therecovery process so that the specific configuration and/or state of thefilesystems may be re-created on FSVM-3. Further, metadata associatedwith the failed CVM-1 110 a, e.g., metadata related to volume groups 368a, 369 a, may be transferred to the alternate CVM (e.g., CVM-2 or CVM-3)that the specific configuration and/or state of CVM-1 may be re-createdon the alternative CVM. When FSVM-1 again becomes available, e.g., afterit has been re-started and has resumed operation on Host-1 200 a oranother host machine, FSVM-1 may assert leadership and take back its IPaddress (10.1.1.1) and storage assignments (FileSystem-1A and FS1-Pool-1366 a, and FileSystem-2A and FS2-Pool-2 366 b) from FSVM-3. When CVM-1again becomes available, MPIO or FSVM-1 may switch the FSVM to CVMcommunication paths (iSCSI paths) for FileSystem-1A 364 a andFileSystem-2A 365 a back to the pre-failure paths, e.g., the paths tovolume groups VG1 368 a and 369 a in CVM-1 110 a, or the selectedalternate path may remain in use. For example, the MPIO configurationmay specify that fail back to FSVM-1 is to occur when the primary pathis restored, since communication between FSVM-1 and CVM-1 is local andmay be faster than communication between FSVM-1 and CVM-2 or CVM-3. Inthis case, the paths between CVM-2 and/or CVM-3 and FSVM-1 may againbecome failover (e.g., standby) paths.

FIGS. 4A and 4B illustrate an example hierarchical namespace 400 of afile server according to particular embodiments. Cluster-1 402 is acluster, which may contain one or more file server instances, such as aninstance named FS1.domain.com 404. Although one cluster is shown inFIGS. 4A and 4B, there may be multiple clusters, and each cluster mayinclude one or more file server instances. The file serverFS1.domain.com 404 contains three shares: Share-1 406, Share-2 408, andShare-3 410. Share-1 may be a home directory share on which userdirectories are stored, and Share-2 and Share-3 may be departmentalshares for two different departments of a business organization, forexample. Each share has an associated size in gigabytes, e.g., 100 Gb(gigabytes) for Share-1, 100 Gb for Share-2, and 10 Gb for Share-3. Thesizes may indicate a total capacity, including used and free space, ormay indicate used space or free space. Share-1 includes three folders,Folder-A1 412, Folder-A2 414, and Folder-A3 416. The capacity ofFolder-A1 is 18 Gb, Folder-A2 is 16 Gb, and Folder-A3 is 66 Gb. Further,each folder is associated with a user, referred to as an owner.Folder-A1 is owned by User-1, Folder-A2 by User-2, and Folder-A3 byUser-3. Folder-A1 contains a file named File-A1-1 418, of size 18 Gb.Folder-A2 contains 32 files, each of size 0.5 Gb, named File-A2-1 420through File-A2-32 422. Folder-A3 contains 33 files, each of size 2 Gb,named File-A3-1 423 and File-A3-2 424 through File-A3-33 426.

FIG. 4B shows the contents of Share-2 408 and Share-3 410 ofFS1.domain.com 404. Share-2 contains a folder named Folder-B1 440, ownedby User-1 and having a size of 100 Gb. Folder-B1 contains File-B1-1 442of size 20 Gb, File-B1-2 444 of size 30 Gb, and Folder-B2 446, owned byUser-2 and having size 50 Gb. Folder-B2 contains File-B2-1 448 of size 5Gb, File-B2-2 450 of size 5 Gb, and Folder-B3 452, owned by User-3 andhaving size 40 Gb. Folder-B3 452 contains 20 files of size 2 Gb each,named File-B3-1 454 through File-B3-20 456. Share-3 contains threefolders: Folder-C7 429 owned by User-1 of size 3 Gb, Folder-C8 430 ownedby User-2 of size 3 Gb, and Folder-C9 432 owned by User-3 of size 4 Gb.

FIG. 4C illustrates distribution of stored data amongst host machines ina virtualized file server according to particular embodiments. In theexample of FIG. 4C, the three shares are spread across three hostmachines 200 a-c. Approximately one-third of each share is located oneach of the three FSVMs 170 a-c. For example, approximately one-third ofShare-3's files are located on each of the three FSVMs 170 a-c. Notethat from a user's point of a view, a share looks like a directory.Although the files in the shares (and in directories) are distributedacross the three host machines 200 a-c, the VFS 202 provides a directorystructure having a single namespace in which client executing on userVMs 101 and 102 may access the files in a location-transparent way,e.g., without knowing which host machines store which files (or whichblocks of files).

In the example of FIG. 4C, Host-1 stores (e.g., is assigned to) 28 Gb ofShare-1, including 18 Gb for File-A1-1 418 and 2 Gb each for File-A3-1423 through File-A3-5 425, 33 Gb of Share-2, including 20 Gb forFile-B1-1 and 13 Gb for File-B1-2, and 3 Gb of Share-3, including 3 Gbof Folder-C7. Host-2 stores 26 Gb of Share-1, including 0.5 Gb each ofFile-A2-1 420 through File-A2-32 422 (16 Gb total) and 2 Gb each ofFile-A3-6 426 through File-A3-10 427 (10 Gb total), 27 Gb of Share-2,including 17 Gb of File-B1-2, 5 Gb of File-B2-1, and 5 Gb of File-B2-2,and 3 Gb of Share-3, including 3 Gb of Folder-C8. Host-3 stores 46 Gb ofShare-1, including 2 Gb each of File-A3-11 429 through File-A3-33 428(66 Gb total), 40 Gb of Share-2, including 2 Gb each of File-B3-1 454through File-B3-20 456, and Share-3 stores 4 Gb of Share-3, including 4Gb of Folder-C9 432.

In particular embodiments, a system for managing communicationconnections in a virtualization environment includes a plurality of hostmachines implementing a virtualization environment. Each of the hostmachines includes a hypervisor and at least one user virtual machine(user VM) 101. The system may also include a connection agent, an I/Ocontroller, and/or a virtual disk comprising a plurality of storagedevices. The virtual disk may be accessible by all of the I/Ocontrollers, and the I/O controllers may conduct I/O transactions withthe virtual disk based on I/O requests received from the user VMs 101.The I/O requests may be, for example, requests to perform particularstorage access operations such as list folders and files in a specifiedfolder, create a new file or folder, open an existing file for readingor writing, read data from or write data to a file, as well as filemanipulation operations to rename, delete, copy, or get details, such asmetadata, of files or folders. Each I/O request may reference, e.g.,identify by name or numeric identifier, a file or folder on which theassociated storage access operation is to be performed. The systemfurther includes a virtualized file server, which includes a pluralityof FSVMs 170 and associated local storage 122. Each FSVM 170 andassociated local storage device 122 is local to a corresponding one ofthe host machines 200. The FSVMs 170 conduct I/O transactions with theirassociated local storage 122 based on I/O requests received from theuser VMs 101. For each one of the host machines 200, each of the userVMs 101 on the one of the host machines 200 sends each of its respectiveI/O requests 383 to a selected one of the FSVMs 170, which may beselected based on a lookup table 360, e.g., a sharding map, that maps afile 318, folder 312, or other storage resource referenced by the I/Orequest to the selected one of the FSVMs 170).

In particular embodiments, the initial FSVM to receive the request fromthe user VM may be determined by selecting any of the FSVMs 170 on thenetwork 140, e.g., at random, by round robin selection, or by aload-balancing algorithm, and sending an I/O request 383 to the selectedFSVM 170 via the network 140 or via local communication within the hostmachine 200. Local communication may be used if the file 318 or folder412 referenced by the I/O request is local to the selected FSVM, e.g.,the referenced file or folder is located on the same host machine 200 asthe selected FSVM 170. In this local case, the I/O request 383 need notbe sent via the network 140. Instead, the I/O request 383 may be sent tothe selected FSVM 170 using local communication, e.g., a localcommunication protocol such as UNIX domain sockets, a loopbackcommunication interface, inter-process communication on the host machine200, or the like. The selected FSVM 170 may perform the I/O transactionspecified in the I/O request and return the result of the transactionvia local communication. If the referenced file or folder is not localto the selected FSVM, then the selected FSVM may return a resultindicating that the I/O request cannot be performed because the file orfolder is not local to the FSVM. The user VM may then submit a REFERRALrequest or the like to the selected FSVM, which may determine which FSVMthe referenced file or folder is local to (e.g., by looking up the FSVMin a distributed mapping table), and return the identity of that FSVM tothe user VM in a REDIRECT response or the like. Alternatively, theselected FSVM may determine which FSVM the referenced file or folder islocal to, and return the identity of that FSVM to the user VM in thefirst response without the REFERRAL and REDIRECT messages. Other ways ofredirecting the user VM to the FSVM of the referenced file arecontemplated. For example, the FSVM that is on the same host as therequesting user VM (e.g., local to the requesting user VM) may determinewhich FSVM the file or folder is local to, and inform the requestinguser VM of the identity of that FSVM without communicating with adifferent host.

In particular embodiments, the file or folder referenced by the I/Orequest includes a file server name that identifies a virtualized fileserver on which the file or folder is stored. The file server name mayalso include or be associated with a share name that identifies a share,file system, partition, or volume on which the file or folder is stored.Each of the user VMs on the host machine may send a host name lookuprequest, e.g., to a domain name service, that includes the file servername, and may receive one or more network addresses of one or more hostmachines on which the file or folder is stored.

In particular embodiments, as described above, the FSVM may send the I/Orequest to a selected one of the FSVMs. The selected one of the FSVMsmay be identified by one of the host machine network addresses receivedabove. In one aspect, the file or folder is stored in the local storageof one of the host machines, and the identity of the host machines maybe determined as described below.

In particular embodiments, when the file or folder is not located onstorage local to the selected FSVM, e.g., when the selected FSVM is notlocal to the identified host machine, the selected FSVM responds to theI/O request with an indication that the file or folder is not located onthe identified host machine. Alternatively, the FSVM may look up theidentity of the host machine on which the file or folder is located, andreturn the identity of the host machine in a response.

In particular embodiments, when the host machine receives a responseindicating that the file or folder is not located in the local storageof the selected FSVM, the host machine may send a referral request(referencing the I/O request or the file or folder from the I/O request)to the selected FSVM. When the selected FSVM receives the referralrequest, the selected FSVM identifies one of the host machines that isassociated with a file or folder referenced in the referral requestbased on an association that maps files to host machines, such as asharding table (which may be stored by the centralized coordinationservice). When the selected FSVM is not local to the host machine, thenthe selected FSVM sends a redirect response that redirects the user VMon the host machine to the machine on which the selected FSVM islocated. That is, the redirect response may reference the identifiedhost machine (and by association the selected second one of the FSVMs).In particular embodiments, the user VM on the host machine receives theredirect response and may cache an association between the file orfolder referenced in the I/O request and the host machine referenced inthe redirect response.

In particular embodiments, the user VM on the host machine may send ahost name lookup request that includes the name of the identified hostmachine to a name service, and may receive the network address of theidentified host machine from the name service. The user VM on the hostmachine may then send the I/O request to the network address receivedfrom the name service. The FSVM on the host machine may receive the I/Orequest and performs the I/O transaction specified therein. That is,when the FSVM is local to the identified host machine, the FSVM performsthe I/O transaction based on the I/O request. After performing orrequesting the I/O transaction, the FSVM may send a response thatincludes a result of the I/O transaction back to the requesting hostmachine. I/O requests from the user VM may be generated by a clientlibrary that implements file I/O and is used by client program code(such as an application program).

FIG. 5 illustrates an example method for accessing data in a virtualizedfile server according to particular embodiments. The client system 330may access the data, such as a specified folder, as follows. At step502, the client system 330 receives a storage access request from anapplication executing in a user VM. Each storage access requestreferences a file path (e.g., \\FS1.share.com\share-1\Folder-1), whichincludes a file or folder name and further includes or can be used toidentify a share name (e.g., FS1.share.com\share-1) or an NFS remotefile system name (e.g., fs1.share.com:/share-1. The storage accessrequest may also include an operation type (e.g., read, write, delete,rename, etc.), a position in the file (for read/write requests), data tobe written (for write requests), quantity of data to be read (for readrequests), a new file path (for rename requests), folder name (forfolder creation requests) or other information appropriate for theoperation type. At step 504, the client system may send a DNS queryrequest for the file server portion of the share name (e.g.,\\fs1.domain.com for the share \\FS1.domain.com\share-1) to a nameserver 332, which may return the identity of a selected host machine asa result. The name server 332 may be a DNS server. The selected hostmachine is not necessarily the host machine on which the file or folderitself is located, however, since the share may be distributed amongstmultiple host machines, one of which actually stores the file or folder.In particular embodiments, a FSVM each host machine can determine whichhost machine a file is stored on, and, if a FSVM receives a request fora file stored on a different host machine, the FSVM sends a referralresponse that includes the identity of the host machine on which thefile is stored.

At step 506, the name server 332 may respond to the client with an IP(network) address of one or more host machines on FSVMs for the file orfolder may be located. For example, the DNS server entry FS1.domain.comincludes entries for FSVM-1, FSVM-2, and FSVM-3, which are,respectively, ip-addr1, ip-addr2, ip-addr3 (or 10.1.1.1, 10.1.1.2,10.1.1.3). One of these three example IP addresses may be selected bythe DNS server and returned in a response. In one example, the DNSserver returns the three IP addresses in a different permutation foreach request using DNS round robin so that a different server may beselected by the client for each request to balance the request loadamong the three servers. In this example, ip-addr1 (10.1.1.1) is thefirst address in the list sent in the reply to the client 330, and so isselected by the client as the address to which the I/O request will, atleast initially, be sent. At step 508, the client may send the I/Orequest to access the folder “Folder-3” to the FSVM located on the hostmachine having address ip-addr1. The I/O request may be, e.g., a DFSattach or connect request, an NFS open request, or the like.

At step 510, FSVM-1 170 a on Host-1 200 a receives the I/O request andconsults a map or lookup table, such as the sharding map 360 a, todetermine whether the folder “Folder-3” is stored on a locally-attachedstorage resource of the host machine on which FSVM 170 a is located. Ifso, FSVM 170 a performs executes step 567 to perform the I/O transactionidentified by the I/O request. If not, at step 512 FSVM-1 170 a respondsto the client 330 with an indication that the folder is not located onthe FSVM-1 170 a's host machine 200 a. The indication may be, e.g., aPATH_NOT_COVERED DFS response. At step 514, upon receiving theindication that the file is not located on the FSVM 170 a to which therequest was sent, the client 330 sends a DFS REFERRAL request to FSVM170 a, requesting a referral to the FSVM on which “Folder-3” is stored.At step 545, FSVM 170 a receives the REFERRAL request and sends a DFS“REDIRECT to FSVM-3” response back to the client 330. FSVM 170 a looksup the FSVM on which the folder “Folder-3” is stored in the map 360 athat associates files or shares with host machines. The result of thelookup, FSVM-3 170 c, may have been determined previously by the lookupat step 510 when the initial request for Folder-3 was received, or maybe determined at step 516 when the referral request for Folder-3 isreceived. For example, the map 360 a may be stored in a shared datastructure provided by the centralized coordination service, and thelookup may be performed by accessing the shared data structure. In thisexample, the file or folder is “Folder-3” and map indicates that thefolder is associated with FSVM 170 c, so at step 516 FSVM 170 a may senda REDIRECT response to the client indicating that the requested folderis stored on host machine 200 c (on which FSVM 170 c is located). TheREDIRECT response may reference the host machine 200 c, the FSVM 170 c,the network address of host machine 200 c (e.g., ip-addr3, in which casesteps 518 and 520 may not be necessary), or other identifier for thelocation of the requested folder. The client 330 may receive theREDIRECT response and cache the association between Folder-3 and hostmachine 200 c (and/or FSVM 170 c) for potential future use.

At step 518, the client 330 may send a DNS query request to the DNSserver 332 to determine the IP address of the FSVM specified in thereceived REDIRECT response, which is FSVM 170 c having IP addressip-addr3 in this example. At step 520, the DNS server 332 may send areply to the client 330 indicating the IP address of the requested hostmachine. For example, the reply may be ip-addr3 (or 10.1.1.3), which isthe IP address of FSVM 170 c. At step 522, the client sends the I/Orequest to access Folder-3 to the IP address received in the DNS reply(e.g., ip-addr3). At step 524, the FSVM 170 c on host machine 200 creceives the I/O request that references Folder-3 and looks up Folder-3in the sharding map. At step 526, FSVM 170 c performs the requested I/Otransaction for Folder-3, e.g., by accessing local storage 122 c, andsends the results of the access, e.g., details about Folder-3 in thisexample, such as a list of files and associated metadata, back to theclient 330 in an I/O response. The client 330 receives the I/O responseand may pass the results of the I/O transaction to the application orother program code that requested the access. Any subsequent requestsfor the same data (Folder-3 in this example) by the client 330 may besent directly to host machine 200 c on which the data is stored becausethe client 330 may use the cached identity of the host machine or FSVMon which the data is stored. Although data contained in a folder isaccessed in the example of FIG. 5, other types of data may be accessedsimilarly, e.g., data contained in files.

In particular embodiments, a virtual appliance may be deployed by usinga hypervisor-agnostic bootable image in a variety of hypervisorenvironments. The image may be provided for download from a portalwebsite to a hypervisor storage container. The hypervisor-agnostic imagemay then be converted into a native image that is specific to thehypervisor platform. Using the native image, VMs are then created andconfigured on the hypervisor, network and storage configuration iscompleted. Finally, the VMs may be grouped into a cluster in order toprovide a virtualized service in the form of an appliance. Thehypervisor-agnostic bootable image itself may have been compressed andprepended with headers providing instructions regarding how to performconversion of the hypervisor-agnostic bootable image into the nativeimage.

In particular embodiments, a VFS 202 consists of multiple compute units,e.g., FSVMs 170. These FSVMs 170 act as a single VFS 202 to the outsideworld. Clusters with appropriate platforms and licenses use ahypervisor-agnostic code-image for the VFS 202. This image may be storedas part of pre-created, ready to use disk images. When a user withcorrect privileges decides to create a VFS 202, the image may be clonedN times, where N is the number of FSVMs 170, and the FSVMs 170 arecreated. The FSVMs 170 form a cluster, which may provide a VFS 202 tothe outside world. In this way, the user is abstracted from the complexprocess of deploying the VFS 202, as the input requested from the useris a small number of parameters that can be provided by the user in asimple user interface. The pre-created fileserver image may reduce thedeployment time to be as fast as booting the host machines 200.

In particular embodiments, the VFS comprises multiple FSVMs 170. Thehost machines 200 may combine to act as VFS 202 to the outside world.Each host machine 200 may have two types of vdisks: code and data. Theoperating system (OS) code and file server code reside on the codevdisk. The fileserver persistent data and configuration are stored onthe data vdisk. In a first technique for upgrading the VFS 202, beforethe upgrade process is started, the newer version of the code disk isprepared and cloned N times (where Nis the number of FSVMs 170). Whileupgrading the VFS 202 to the latest version, a new code disk is swappedwith the existing code disk for each FSVM 170. After rebooting the FSVM170, it will be running with newer code, and continues serving the datausing the newer code.

In particular embodiments, in a second technique for upgrading the VFS202, before the upgrade process is started and after the newer versionof the code disk is prepared and cloned, a first FSVM 170 a acquires anupgrade token, swaps the old code disk with the newer disk, and reboots.When the first FSVM 170 a comes back up and is running, the upgradetoken is passed to the next FSVM 170 b, which may perform the swap andreboot, and pass the upgrade token to the next FSVM 170 c. Theseoperations are repeated until the last FSVM 170, e.g., FSVM 170 c inthis example, is upgraded. During the time that each FSVM 170 b is beingrebooted, one of the peer FSVMs 170 a takes over the storage and IPaddress of the FSVM 170 b so that the client does not see anyinterruption in the file service.

In particular embodiments, users dealing with Virtual Disk Image (“VDI”)files and their corresponding root directories are, by definition, boundto their VMs. This binding provides a user VM 101 to root-leveldirectory mapping. A sharding algorithm may determine a mapping betweena user VM 101 and its corresponding host machine 200. This mapping mayin turn provide a root-level directory-to-host machine mapping. Thesharding algorithm may use this mapping and add metadata to keep storageunits and compute units local, e.g., located on, the same host machine.On migration of the user virtual machines 102, metadata and storage willbe moved accordingly.

Particular embodiments may provide enhanced performance via adaptivedata and compute-unit migration. Particular embodiments may provide theability to restrict compute units and storage units to a locationgoverned by user policy.

In particular embodiments, data migration from an existing VFS 202 to anew VFS 202 may be bounded by the speed of connection between theexisting infrastructure (e.g., host machines 200) and the new system(e.g., other host machines). By using smart data ingestion, datamigration speed can be increased with a multiplier of the number of fileserver host machines.

In previous approaches, data is migrated using a utility to copy datafrom one source to one target location. Migration speed is limited bythe connection speed. In particular embodiments, using the smart dataingestion approach described herein, top-level directories in anexisting VFS 202 are preprocessed to acquire the destination hostmachine 200 on a new (destination) VFS 202. When data migration begins,each host machine 200 in the VFS 202 starts data migration with theshare directories assigned, which speeds up data migration with amultiplier of host machine count. By taking advantage of the distributednature of the VFS 202, data migration is performed in parallel to speedup the migration process. Using the same sharding algorithm as file I/Oto decide the migration target ensure the consistency of migrated dataplacement in, e.g., the new (destination) VFS 202. In particularembodiments, no further processing is needed after data is migrated, anddata is ready to be served.

In a first example, when the organization that manages a virtualizedfile server instance (VFS) 202 decides to, for example, segregate theexisting VFS 202 to a departmental level, the VFS 202 may be split intomultiple virtualized file server instances (VFSs) 202 without affectingthe stored data, with zero to minimal down time, and with zero datacopying or migration. In a second example, when an organization thatmanages multiple VFSs 202 decides to merge them into one manageable VFS202, the multiple VFSs 202 may be merged together without affecting thestored data, and with zero to minimal down time and zero data copying ormigration. When an organization needs to merge multiple VFSs 202, then asystem administrator may deploy a new VFS 202 and migrate the storeddata from the multiple VFSs 202 to the newly deployed VFS 202, whichtakes more time and resources. When the organization needs to split theVFS 202, then a system administrator may deploy new VFSs 202 and migratethe data from the old VFS 202 to the newly deployed VFSs 202, which alsotakes more time and resources.

In particular embodiments, the splitting and merging operations may beperformed as follows. To split an existing VFS 202, e.g., upon a systemadministrator's request, the following operations may be performed:

-   -   1. Select the FSVMs 170 to be segregated from VFS 202.    -   2. The FSVMs 170 are removed one by one.    -   3. Before removing a FSVM 170 from the VFS 202, first select a        lightly loaded FSVM 170 and voluntarily relinquish the storage        resources to the selected FSVM 170. The IP address of the FSVM        170 being removed may also be moved to the selected FSVM 170 to        retain SMB client connections.    -   4. After removing all the FSVMs 170 from the VFS 202, a new VFS        202 is constructed.    -   5. The FSVMs 170 of the new VFS 202 join the domain and start        serving the new shares. Old shares may still be served by the        old VFS 202. Once the administrator decides to move the old        shares to the new VFS 202, trigger a storage transition that        relinquishes the storage to the appropriate selected FSVMs 170        and move the IP addresses of FSVMs 170 of the old VFS 202 to        FSVMs 170 of the new VFS 202.    -   6. The same process may be continued to segregate other VFSs        202.

In particular embodiments, to merge multiple VFSs 202 together, e.g.,upon a system administrator's request, an election is triggered betweenthe multiple VFSs 202 based on the virtual IP address or based onpreference policies. The VFS 202 that wins the election or is selectedby an administrator is treated as a master VFS. All other VFSs then jointo the master VFS's ACTIVE DIRECTORY domain. FSVMs 170 from all slaveVFSs 202 may be added to the master VFS 202, and storage pool metadataof the slave VFSs 202 is modified to serve for the master VFS 202. Thefollowing operations may be performed to merge the slave VFSs into themaster VFS:

-   -   1. Select the VFSs 202 to be merged.    -   2. Initiate the election to elect the master VFS 202 based on        the policy of the VFS 202 that has the higher IP address.    -   3. Once the master VFS has been selected, clients connect to it.    -   4. Select a slave VFS to merge.    -   5. Relinquish the storage to a lightly-loaded FSVM on the master        VFS and move the IP address to refer to the lightly-loaded FSVM.    -   6. Start serving SMB clients for new and old shares.    -   7. Stop the slave file server, add its FSVM(s) one by one to the        new master file server, and take back its resource on the new        master file server.    -   8. Continue these steps for other slave file servers.

In scenarios such as a company splitting into multiple companies, itcould be a requirement that a single VFS 202 is split into two VFSs 202.However, there may be certain SAMBA shares in the original VFS 202 thatneed to be made accessible to both the VFSs 202. As an example, considertwo different fileservers FS1 and FS2. FS1 originally hosted a shareshare1′. FS2 needs the ability to read/write to the share ‘share1’. TheSMB requests for ‘share1’ on FS2 may be forwarded or proxied to FS1,thereby allowing the share ‘share1’ to be readable/writable from twodifferent VFSs 202. Another approach is to NFS-mount the original shareon the new VFS 202, to provide a single namespace. The ability toselectively choose certain shares to be shared across other VFSs 202ensures a tight security boundary at the VFS level, along with thecollaborative access via two different VFSs.

In particular embodiments, disaster recovery of the VFS 202 may beperformed by making backups and replicating delta changes in a storagelayer, then recovering the data stored in the VFS 202 at a remote site.The data may be recovered by reconstructing the VFS 202 from areplicated configuration. In a production environment, the data storedon a VFS 202 is securely protected and restored on a remote locationwithout loss of the data and metadata within a supported Recovery PointObjective (which may be the age of files to be recovered from backupstorage for normal operations to resume). A custom replication policymay be configured for the VFS 202, and the ability may be provided tomap the VFS 202's configuration between sites to provide disasterrecovery of virtual file-services across geographical locations.Particular embodiments may provide the ability to protect individualshares or share groups by protecting the volume group(s) used forfile-services storage, e.g., by adding them to a protection domain.Users may apply the replication and backup policies on the protectiondomain to configure the Recovery Point Objective, recovery sites(alternate cluster or cloud), and replication constraints such asbandwidth and schedule. Particular embodiments may take lightweightsnapshots and transfer the delta of the snapshots for the given volumegroups. Along with file-services share data, particular embodiments mayalso transfer the VFS configuration e.g. file-server size, compute-unitconfiguration, and metadata, e.g., share ACLs, quotas, and so on.Particular embodiments may also provide a simplified user interface toconfigure mapping of network, DNS-servers, active-directory, etc.between remote sites. Potential benefits may include:

-   -   1. Granular level of protection (share or group of shares) to        configure different Recovery Point Objective.    -   2. Custom replication policies to utilize the network resources        effectively for replication.    -   3. Fine control on network and ecosystem resource mapping        between sites.    -   4. Light weight snapshot includes share data delta, metadata and        file-server configuration leading to less replication traffic        across sites.    -   5. One click restore of file-services on remote site.    -   6. Distribution of share replication across multiple remote        sites.    -   7. Multiple recovery points on multiple remote sites for        multi-site failures.

FIG. 6 illustrates an example of how a file server ‘FS1’ may be deployedacross multiple clusters according to particular embodiments. Particularembodiments may facilitate deploying and managing a VFS 202 whosenetworking, compute-unit, and storage resources are distributed acrossmultiple clusters from a single management portal. Particularembodiments may create a VFS 202 and distribute compute units, which maybe the FSVMs 170. A portal user interface may be used by a user oradministrator to create a VFS 202. While creating the VFS 202, a user isgiven a list of clusters that may be used to distribute the computeunits (e.g., FSVMs, or may perform the operations of FSVMs as describedherein), networking (IP addresses) and storage (containers). In theexample of FIG. 6, the user has chosen three clusters, Cluster 1,Cluster 2, and Cluster 3. Three FSVMs are created on each cluster, for atotal of 9 FSVMs across the three clusters. Each cluster for this fileserver hosts a separate container, which holds a part of the file serverdata. The containers are labeled Container 1, Container 2, and Container3. The containers are hidden from the user.

Particular embodiments may create shares and distribute storage unitsand compute units. The portal user interface may be used to create ashare ‘share1’ within the file server FS1. The data within ‘share1’ isdistributed across all the clusters. A storage pool of multiple vDisksis constructed on all FSVMs across all clusters. Each ZPool on each FSVMis responsible for a subset of the ‘share1’ data. The share is shardedat the top-level directories across FSVMs residing in differentclusters. The sharding strategy is as follows. Assuming that directoriesdir1, dir2, dir3, dir4, dir5, dir6 have been created:

-   -   1. Each FSVM within each cluster hosts a storage pool created        from a subset of the container storage. A background process        periodically runs on a leader FSVM in each cluster to aggregate        the File system space used for each share across all FSVMs in        the cluster. This data is published to a cluster manager that        stores the data in an entity database, e.g., APACHE CASSANDRA or        the like. The cluster manager may be NUTANIX PRISM CENTRAL,        which is a multi-cluster manager responsible for managing        multiple clusters to provide a single, centralized management        interface.    -   2. User creates a new top-level directory, e.g., ‘dir7’.    -   3. The Samba VFS layer intercepts the directory creation request        and consults a database to determine whether the directory is        hosted by any FSVM (or FSVM). If it is not, the VFS layer makes        an RPC call to a file server service running in PRISM CENTRAL to        identify a location (which may be an optimal location) for        ‘dir7’.    -   4. The file server service running in PRISM CENTRAL retrieves        the per-cluster usage statistics for each share that it received        in step 1 above, and chooses the cluster that has the least used        space for the share ‘share1’. In the example of FIG. 6, Cluster        1 is chosen. The file server service may also provide an option        to simply choose the cluster that has the greatest amount of        free fileserver container space.    -   5. Next, the file server service running in PRISM CENTRAL        queries Cluster 1 for average CPU utilization for the past 24        hours for all VMs within Cluster 1. The file server service then        chooses the least loaded FSVM. The file server service in PRISM        CENTRAL returns this <Cluster 1, FSVM2 FQDN> tuple back to the        VFS layer.    -   6. The VFS layer now knows the FSVM2 FQDN, which should host        ‘dir7’ and hence creates this new directory on the Unix        file-system corresponding to FSVM2. The VFS layer records this        mapping <Share1, dir1>→Cluster 1, FSVM2 in a database, and        returns a PATH_NOT_COVERED message to the client.    -   7. Through DFS referral, the SAMBA client requests the path for        the directory ‘dir1’. The FSVM looks up ‘dir1’ in the database,        and returns FSVM2 IP to the client. The client now accesses on        the FSVM2 file system.

The file system on any FSVM may be composed of vDisks. Since vDisks aredistributed across the cluster, this arrangement provides uniformsharding of storage within the cluster. The sharding strategy describedabove causes all clusters' containers and FSVMs to be used, and achievesuniform sharding of storage units and compute units across the clusters.

Particular embodiments may provide cluster-aware sharding andcluster-aware share level quotas. At the time of share creation, user isgiven the option to co-locate the data for the share within certainclusters. This option may be useful if the user wishes to have one setof shares distributed within a certain geographical boundary, and adifferent set of shares distributed across a different geographicalboundary, in which case the above sharding strategy remains the same. Instep 4 above, only those clusters that were selected while creating theshare would be made available to be considered for sharding. Thistechnique provides cluster-aware sharding.

Similarly, quotas can be set on a file server service 202. Quotas mayset a limit on the amount of data storage to be used for each sharewithin each cluster. Since file server service stores a per-share,per-cluster storage usage, it can detect when a cluster-level sharequota is reached. Depending on the quota policy, the user may be alertedwhen this cluster-level quota is reached, or the file server service maynotify the FSVM (or FSVM) leader within the cluster whose quota has beenreached via RPC. On receiving this notification, the FSVM leader maymake all filesystems for that share across FSVMs read-only to respectthe storage quota limit.

Particular embodiments may handle sharding by considering geographicquotas, user-based quotas, fault tolerance of clusters, availableresources across clusters, etc. Some benefits may include:

-   -   1. Provides uniform auto sharding of compute, network, and        storage units across multiple clusters, which also leads to a        smaller fault domain    -   2. File server centrally managed from a single interface,        although resources are distributed, leading to easy        manageability.    -   3. Provides flexibility of co-locating shares where necessary        and distributing them across clusters when necessary.    -   4. Provides ability to set cluster aware share level quotas,        which could be utilized for location-aware sharding. and    -   5. Fault tolerant within cluster and capable of tolerating        entire cluster failure.

Particular embodiments may identify corrupted or infected data andrecover a consistent version of the data from a VFS 202. When user datais infected by a virus or corrupted by a file system or storage system,identifying the corrupted data and the needed recovery level may bedifficult. If the appropriate recovery level is not detected and data isrecovered at the wrong place, then a valid version of data may be lost.Particular embodiments may provide capabilities to virtual file servicesto detect problems from file level to storage level. Systemadministrators need not worry about detecting and recovering aconsistent version of data when the system administrator detects thecorruption and infected data and manually recovers the data from a filesystem or from storage system. A self-healing mechanism of the VFS 202frequently takes snapshots of file system and storage pools and monitorsthe user data at file system and storage system levels. In particularembodiments, a virtualized file server may accomplish the followinglevels of detection and recovery:

-   -   1. File/folder level recovery: File system or anti-virus or        other internal modules can detect the file or folder level        infection or corruption. Self-Healing mechanism monitors these        events and once it detects, it will recover those particular        data from the previous file system snapshot by overwriting the        infected/corrupted files/folders.    -   2. File System level recovery: Self-healing mechanism monitors        the checksum of the file system and if it finds any discrepancy        on that, it will recover the file system with its latest        snapshot.    -   3. Storage level recovery: Self-Healing mechanism monitors        storage-pool corruption and alerts generated by the cluster and        detect the data loses and corruption. Once it detects the data        corruption/data loss, it will recover the storage-pool for the        latest snapshot.

Distributed Self-Healing: Since virtualized file server compute andstorage units are distributed across multiple host machines 200, theself-healing mechanism efficiently monitors the corruptions and dataloss in parallel and distributed fashion on all the host machines anddetects and recovers that particular data without affecting the overallfile server 202.

Some benefits may include: Detection of and recovery from data loss,data corruption and infected files on file/folder level, file systemlevel and storage level without manual intervention. Efficient detectionof and recovery from the data loss, data corruption and infected filesin parallel and distributed fashion. Recovery from data loss and datacorruption without affecting the overall file server 202.

Particular embodiments may back up cold data stored in a cluster to anobject store, which is either in a public cloud (e.g., AMAZON WEBSERVICES), or to a low-cost storage medium within the same cluster.Particular embodiments may then retrieve the backed-up volume groups asneeded to restore files for the file server. Particular embodiments mayprovide a method to backup data on a virtualized file server running ona hyper-converged infrastructure to a low-cost storage medium hosted onthe same physical infrastructure. This consists of a virtualized serverrunning on the same hyper-converged infrastructure providing an objectstore interface (such as AMAZON S3) with storage as low-cost media suchas SMR drives. This particular virtual machine can act as a backupserver for other VMs running on the same infrastructure.

Particular embodiments of the backup server may be hosted on the samehyper converged infrastructure as the compute and storage. Particularembodiments of the backup server may be used for low cost storage medialike SMR drives attached to the same hyper converged infrastructure.Particular embodiments of the backup server may provide generic objectstore interfaces such as AMAZON S3. Particular embodiments of the backupserver may provide the same level of availability as the other highlyavailable services (such as FSVM) run on the cluster.

Particular embodiments may include a cloud service as a storage tier ofa virtualized file server service. Particular embodiments may thenretrieve the backed-up volume groups as needed to restore files for thefile server.

Particular embodiments may provide block awareness for a virtualizedfile server service in order to maintain availability of virtual fileserver services in case of block failure by deploying FSVMs 170 ondifferent host machines 200. In case of block failure (e.g., due topower loss affecting a block, a.k.a. hardware host machine), thehigh-availability features attempt to migrate the VMs running on thosehost machines to available running host machines. If there are notenough resources on the available running host machines, then thefile-server HA features are triggered, and online FSVM(s) take ownershipof the volume-group of offline FSVM(s). When one FSVM with metadataservice is down, the file-server may continue to serve requests to endusers without any performance degradation. Potential benefits mayinclude:

-   -   1. Virtualized file-server is available even if one block in the        cluster goes down.    -   2. Users or administrators need not to reserve the resources or        free up the resources for block failure to get the virtualized        file-server working.    -   3. In a hyper-converged deployment, the user VMs can be        prioritized over FSVMs for migration during block failure.

Particular embodiments may recover from multimode file service failuresin a scale-out NAS environment with minimal down time. Traditional fileserver deployments protected against single host machine failures byhaving a standby host machine. Detection of service failures is notspontaneous and issues can occur with keeping backup host machinessynchronized. Further, if the new active host machine is also down,there may be no way to recover the service. These issues not only causeservice interruption but also create complications if there are multiplehost machine failures.

In a scale out NAS environment, host machine and service failures may bedetected and recovered from without interrupting service to clients.Using distributed cluster health service, these service failures may bedetected, and the other active host machines may take over the failedhost machine services (both storage and network). Each host machine incluster acts as potential backup host machine, which will help withmanaging multiple simultaneous host machine failures based on clusterhealth. So, even if the new active host machine is down, other hostmachines in the cluster can take over the new active host machine's loadand provide continuous availability. In this way, clients using thescale-out NAS services do not see any downtime or service interruptionswhen multiple host machines in the cluster are down.

Particular embodiments may help to avoid catastrophic failures due toresource exhaustion. In scenarios such as a user's home directory beingaccessed for read/write operations, the user may not be able todetermine how much disk quota is assigned to the user or how much actualspace is available to the user to write data.

As an example, consider a scenario when many users have their homedirectories on the same share. Existing technologies display the user'sdrive size as being the same as total share size, thereby giving theuser the perception that the total share size is available to writedata. However, when the user's quota limit has been met or exceeded, anywrite to the drive fails.

Particular embodiments may expose user-specific data so that when eachuser accesses their VDI environment they see their home directory as amounted drive, and see data specific to their disk portion, such as diskcapacity, average rate of utilization of space, frequency of diskaccesses, file type, etc. On every soft quota limit reached, the usermay be alerted through email that they are about to exhaust their diskquota. Less-frequently-accessed files, folders, and other items may bearchived automatically to the cloud.

Particular embodiments may provide high availability of storage servicesin a scale out file-server. In traditional file server deployments,high-availability is supported by configuring host machines as pairswhere the storage resources are inter-connected between two hostmachines. So, if one of the host machines fails, the other host machinein the pair may take over the storage resources along with the IPaddress. One limitation with this approach is that an even number ofhost machines is needed in the cluster. In a scale out file-server,minimal to zero disruption occurs in case of any failure. In avirtualized scale-out file server, all host machines in the scale outcluster monitor the health for every other host machine. If one of thehost machines experiences down time because of either a planned shutdownor unplanned host machine failures, one of the host machines startstaking over the storage resources of the down host machine. At the sametime, the IP address fails over so that clients can continue to contactthe takeover host machine without any disruptions. To increase the loadbalancing, the failover storage resources may be distributed to multiplehost machines, so that the down host machine resources may bedistributed across different host machines.

FIG. 7 is a block diagram of an illustrative computing system 700suitable for implementing particular embodiments. In particularembodiments, one or more computer systems 700 perform one or more stepsof one or more methods described or illustrated herein. In particularembodiments, one or more computer systems 700 provide functionalitydescribed or illustrated herein. In particular embodiments, softwarerunning on one or more computer systems 700 performs one or more stepsof one or more methods described or illustrated herein or providesfunctionality described or illustrated herein. Particular embodimentsinclude one or more portions of one or more computer systems 700.Herein, reference to a computer system may encompass a computing device,and vice versa, where appropriate. Moreover, reference to a computersystem may encompass one or more computer systems, where appropriate.

This disclosure contemplates any suitable number of computer systems700. This disclosure contemplates computer system 700 taking anysuitable physical form. As example and not by way of limitation,computer system 700 may be an embedded computer system, a system-on-chip(SOC), a single-board computer system (SBC) (such as, for example, acomputer-on-module (COM) or system-on-module (SOM)), a desktop computersystem, a mainframe, a mesh of computer systems, a server, a laptop ornotebook computer system, a tablet computer system, or a combination oftwo or more of these. Where appropriate, computer system 700 may includeone or more computer systems 700; be unitary or distributed; spanmultiple locations; span multiple machines; span multiple data centers;or reside in a cloud, which may include one or more cloud components inone or more networks. Where appropriate, one or more computer systems700 may perform without substantial spatial or temporal limitation oneor more steps of one or more methods described or illustrated herein. Asan example and not by way of limitation, one or more computer systems700 may perform in real time or in batch mode one or more steps of oneor more methods described or illustrated herein. One or more computersystems 700 may perform at different times or at different locations oneor more steps of one or more methods described or illustrated herein,where appropriate.

Computer system 700 includes a bus 702 (e.g., an address bus and a databus) or other communication mechanism for communicating information,which interconnects subsystems and devices, such as processor 704,memory 706 (e.g., RAM), static storage 708 (e.g., ROM), dynamic storage710 (e.g., magnetic or optical), communication interface 714 (e.g.,modem, Ethernet card, a network interface controller (MC) or networkadapter for communicating with an Ethernet or other wire-based network,a wireless NIC (WNIC) or wireless adapter for communicating with awireless network, such as a WI-FI network), input/output (I/O) interface712 (e.g., keyboard, keypad, mouse, microphone). In particularembodiments, computer system 700 may include one or more of any suchcomponents.

In particular embodiments, processor 704 includes hardware for executinginstructions, such as those making up a computer program. As an exampleand not by way of limitation, to execute instructions, processor 704 mayretrieve (or fetch) the instructions from an internal register, aninternal cache, memory 706, static storage 708, or dynamic storage 710;decode and execute them; and then write one or more results to aninternal register, an internal cache, memory 706, static storage 708, ordynamic storage 710. In particular embodiments, processor 704 mayinclude one or more internal caches for data, instructions, oraddresses. This disclosure contemplates processor 704 including anysuitable number of any suitable internal caches, where appropriate. Asan example and not by way of limitation, processor 704 may include oneor more instruction caches, one or more data caches, and one or moretranslation lookaside buffers (TLBs). Instructions in the instructioncaches may be copies of instructions in memory 706, static storage 708,or dynamic storage 710, and the instruction caches may speed upretrieval of those instructions by processor 704. Data in the datacaches may be copies of data in memory 706, static storage 708, ordynamic storage 710 for instructions executing at processor 704 tooperate on; the results of previous instructions executed at processor704 for access by subsequent instructions executing at processor 704 orfor writing to memory 706, static storage 708, or dynamic storage 710;or other suitable data. The data caches may speed up read or writeoperations by processor 704. The TLBs may speed up virtual-addresstranslation for processor 704. In particular embodiments, processor 704may include one or more internal registers for data, instructions, oraddresses. This disclosure contemplates processor 704 including anysuitable number of any suitable internal registers, where appropriate.Where appropriate, processor 704 may include one or more arithmeticlogic units (ALUs); be a multi-core processor; or include one or moreprocessors 702. Although this disclosure describes and illustrates aparticular processor, this disclosure contemplates any suitableprocessor.

In particular embodiments, I/O interface 712 includes hardware,software, or both, providing one or more interfaces for communicationbetween computer system 700 and one or more I/O devices. Computer system700 may include one or more of these I/O devices, where appropriate. Oneor more of these I/O devices may enable communication between a personand computer system 700. As an example and not by way of limitation, anI/O device may include a keyboard, keypad, microphone, monitor, mouse,printer, scanner, speaker, still camera, stylus, tablet, touch screen,trackball, video camera, another suitable I/O device or a combination oftwo or more of these. An I/O device may include one or more sensors.This disclosure contemplates any suitable I/O devices and any suitableI/O interfaces 712 for them. Where appropriate, I/O interface 712 mayinclude one or more device or software drivers enabling processor 704 todrive one or more of these I/O devices. I/O interface 712 may includeone or more I/O interfaces 712, where appropriate. Although thisdisclosure describes and illustrates a particular I/O interface, thisdisclosure contemplates any suitable I/O interface.

In particular embodiments, communication interface 714 includeshardware, software, or both providing one or more interfaces forcommunication (such as, for example, packet-based communication) betweencomputer system 700 and one or more other computer systems 700 or one ormore networks. As an example and not by way of limitation, communicationinterface 714 may include a network interface controller (NIC) ornetwork adapter for communicating with an Ethernet or other wire-basednetwork or a wireless NIC (WNIC) or wireless adapter for communicatingwith a wireless network, such as a WI-FI network. This disclosurecontemplates any suitable network and any suitable communicationinterface 714 for it. As an example and not by way of limitation,computer system 700 may communicate with an ad hoc network, a personalarea network (PAN), a local area network (LAN), a wide area network(WAN), a metropolitan area network (MAN), or one or more portions of theInternet or a combination of two or more of these. One or more portionsof one or more of these networks may be wired or wireless. As anexample, computer system 700 may communicate with a wireless PAN (WPAN)(such as, for example, a BLUETOOTH WPAN), a WI-FI network, a WI-MAXnetwork, a cellular telephone network (such as, for example, a GlobalSystem for Mobile Communications (GSM) network), or other suitablewireless network or a combination of two or more of these. Computersystem 700 may include any suitable communication interface 714 for anyof these networks, where appropriate. Communication interface 714 mayinclude one or more communication interfaces 714, where appropriate.Although this disclosure describes and illustrates a particularcommunication interface, this disclosure contemplates any suitablecommunication interface.

One or more memory buses (which may each include an address bus and adata bus) may couple processor 704 to memory 706. Bus 702 may includeone or more memory buses, as described below. In particular embodiments,one or more memory management units (MMUs) reside between processor 704and memory 706 and facilitate accesses to memory 706 requested byprocessor 704. In particular embodiments, memory 706 includes randomaccess memory (RAM). This RAM may be volatile memory, where appropriate.Where appropriate, this RAM may be dynamic RAM (DRAM) or static RAM(SRAM). Moreover, where appropriate, this RAM may be single-ported ormulti-ported RAM. This disclosure contemplates any suitable RAM. Memory706 may include one or more memories 706, where appropriate. Althoughthis disclosure describes and illustrates particular memory, thisdisclosure contemplates any suitable memory.

Where appropriate, the ROM may be mask-programmed ROM, programmable ROM(PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM),electrically alterable ROM (EAROM), or flash memory or a combination oftwo or more of these. In particular embodiments, dynamic storage 710 mayinclude a hard disk drive (HDD), a floppy disk drive, flash memory, anoptical disc, a magneto-optical disc, magnetic tape, or a UniversalSerial Bus (USB) drive or a combination of two or more of these. Dynamicstorage 710 may include removable or non-removable (or fixed) media,where appropriate. Dynamic storage 710 may be internal or external tocomputer system 700, where appropriate. This disclosure contemplatesmass dynamic storage 710 taking any suitable physical form. Dynamicstorage 710 may include one or more storage control units facilitatingcommunication between processor 704 and dynamic storage 710, whereappropriate.

In particular embodiments, bus 702 includes hardware, software, or bothcoupling components of computer system 700 to each other. As an exampleand not by way of limitation, bus 702 may include an AcceleratedGraphics Port (AGP) or other graphics bus, an Enhanced Industry StandardArchitecture (EISA) bus, a front-side bus (FSB), a HYPERTRANSPORT (HT)interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBANDinterconnect, a low-pin-count (LPC) bus, a memory bus, a Micro ChannelArchitecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, aPCI-Express (PCIe) bus, a serial advanced technology attachment (SATA)bus, a Video Electronics Standards Association local (VLB) bus, oranother suitable bus or a combination of two or more of these. Bus 702may include one or more buses 706, where appropriate. Although thisdisclosure describes and illustrates a particular bus, this disclosurecontemplates any suitable bus or interconnect.

According to particular embodiments, computer system 700 performsspecific operations by processor 704 executing one or more sequences ofone or more instructions contained in memory 706. Such instructions maybe read into memory 706 from another computer readable/usable medium,such as static storage 708 or dynamic storage 710. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions. Thus, particular embodiments arenot limited to any specific combination of hardware circuitry and/orsoftware. In one embodiment, the term “logic” shall mean any combinationof software or hardware that is used to implement all or part ofparticular embodiments disclosed herein.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto processor 704 for execution. Such a medium may take many forms,including but not limited to, nonvolatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks,such as static storage 708 or dynamic storage 710. Volatile mediaincludes dynamic memory, such as memory 706.

Common forms of computer readable media include, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read.

In particular embodiments, execution of the sequences of instructions isperformed by a single computer system 700. According to other particularembodiments, two or more computer systems 700 coupled by communicationlink 716 (e.g., LAN, PTSN, or wireless network) may perform the sequenceof instructions in coordination with one another.

Computer system 700 may transmit and receive messages, data, andinstructions, including program, i.e., application code, throughcommunication link 716 and communication interface 714. Received programcode may be executed by processor 704 as it is received, and/or storedin static storage 708 or dynamic storage 710, or other non-volatilestorage for later execution. A database 720 may be used to store dataaccessible by the system 700 by way of data interface 718.

Herein, a computer-readable non-transitory storage medium or media mayinclude one or more semiconductor-based or other integrated circuits(ICs) (such, as for example, field-programmable gate arrays (FPGAs) orapplication-specific ICs (ASICs)), hard disk drives (HDDs), hybrid harddrives (HHDs), optical discs, optical disc drives (ODDs),magneto-optical discs, magneto-optical drives, floppy diskettes, floppydisk drives (FDDs), magnetic tapes, solid-state drives (SSDs),RAM-drives, SECURE DIGITAL cards or drives, any other suitablecomputer-readable non-transitory storage media, or any suitablecombination of two or more of these, where appropriate. Acomputer-readable non-transitory storage medium may be volatile,non-volatile, or a combination of volatile and non-volatile, whereappropriate.

Herein, “or” is inclusive and not exclusive, unless expressly indicatedotherwise or indicated otherwise by context. Therefore, herein, “A or B”means “A, B, or both,” unless expressly indicated otherwise or indicatedotherwise by context. Moreover, “and” is both joint and several, unlessexpressly indicated otherwise or indicated otherwise by context.Therefore, herein, “A and B” means “A and B, jointly or severally,”unless expressly indicated otherwise or indicated otherwise by context.

The scope of this disclosure encompasses all changes, substitutions,variations, alterations, and modifications to the example embodimentsdescribed or illustrated herein that a person having ordinary skill inthe art would comprehend. The scope of this disclosure is not limited tothe example embodiments described or illustrated herein. Moreover,although this disclosure describes and illustrates respectiveembodiments herein as including particular components, elements,feature, functions, operations, or steps, any of these embodiments mayinclude any combination or permutation of any of the components,elements, features, functions, operations, or steps described orillustrated anywhere herein that a person having ordinary skill in theart would comprehend. Furthermore, reference in the appended claims toan apparatus or system or a component of an apparatus or system beingadapted to, arranged to, capable of, configured to, enabled to, operableto, or operative to perform a particular function encompasses thatapparatus, system, component, whether or not it or that particularfunction is activated, turned on, or unlocked, as long as thatapparatus, system, or component is so adapted, arranged, capable,configured, enabled, operable, or operative.

What is claimed is:
 1. A method comprising: presenting a singlenamespace of storage items by a virtualized distributed file system, thevirtualized distributed file system including a cluster of host machinesand at least one file server virtual machine (FSVM) on each host machinein the cluster of host machines, the storage items stored in a storagepool distributed across the cluster, the storage pool including anaggregation of local storage devices of the host machines; processing,by the virtualized distributed file system, an access operation directedto one of the storage items in the single namespace, the one of thestorage items stored in the storage pool; and scaling out thevirtualized distributed file system responsive to a load of thevirtualized distributed file system, wherein the scaling out includesusing a file server virtual machine manager to add an additional hostmachine to the cluster of host machines and using the file servervirtual machine manager to deploy an additional FSVM to the additionalhost machine.
 2. The method of claim 1, further comprising performingmanagement operations at the FSVM manager for the virtualizeddistributed file system, the management operations including deployingthe at least one FSVM on each host machine, a number of the FSVMs in thecluster of host machines configured to form the single namespace of thestorage items.
 3. The method of claim 2, wherein the managementoperations further include configuring each of the number of the FSVMswith a predetermined amount of memory and compute resources based onrespective memory and compute resource parameters.
 4. The method ofclaim 1 further comprising: assigning one or more volume groups to eachFSVM in the cluster of host machines, wherein each volume group includesa group of storage devices of the storage pool; and creating file sharesincluding a storage capacity of the local storage devices of the storagepool, wherein each file share is stored on a respective volume group. 5.The method of claim 4 further comprising performing one or moreinput/output (I/O) transactions between each FSVM and the file sharesstored on the assigned volume groups.
 6. The method of claim 4 furthercomprising: reassigning one or more volume groups from a particular FSVMto another FSVM in response to the scaling out of the virtualizeddistributed file system to more evenly distribute the I/O transactionsamong the FSVMs.
 7. The method of claim 1, further comprisingapproximating the load of the virtualized distributed file systemincluding determining resources consumed by a particular FSVM of thevirtualized distributed file system.
 8. The method of claim 7, furthercomprising comparing the resources consumed by the particular FSVM witha capacity threshold for the particular FSVM.
 9. A virtualizeddistributed file system comprising: a cluster of host machines, eachhost machine including at least one file server virtual machine (FSVM)and a local storage device; and a storage pool distributed across thecluster, the storage pool including an aggregation of the local storagedevices of the host machines, wherein the FSVMs are configured topresent a single namespace of storage items stored in the storage pool;wherein the virtualized distributed file system is configured to scaleout responsive to a load of the virtualized distributed file system,wherein the scale out includes adding an additional host machine to thecluster of host machines and deploying an additional FSVM to theadditional host machine.
 10. The virtualized distributed file system ofclaim 9, further comprising a file server virtual machine managerconfigured to perform the scale out.
 11. The virtualized distributedfile system of claim 9, further comprising a file server virtual machinemanager adapted to configure each of the number of the FSVMs with apredetermined amount of memory and compute resources based on respectivememory and compute resource parameters.
 12. The virtualized distributedfile system of claim 9, further comprising: one or more volume groupsassigned to each FSVM in the cluster of host machines, wherein eachvolume group includes a group of storage devices of the storage pool;and one or more file shares including a storage capacity of the localstorage devices of the storage pool, wherein each file share is storedon a respective volume group.
 13. The virtualized distributed filesystem of claim 12 wherein the virtualized distributed file system isfurther configured to perform one or more input/output (I/O)transactions between each FSVM and the one or more file shares stored onthe assigned volume groups.
 14. The virtualized distributed file systemof claim 12 wherein the virtualized distributed file system is furtherconfigured to reassign one or more volume groups from a particular FSVMto another FSVM in response to the scale out of the virtualizeddistributed file system to more evenly distribute the I/O transactionsamong the FSVMs.
 15. The virtualized distributed file system of claim 9,wherein the virtualized distributed file system is further configured toapproximate the load of the virtualized distributed file systemincluding determining resources consumed by a particular FSVM of thevirtualized distributed file system.
 16. The virtualized distributedfile system of claim 15, wherein the virtualized distributed file systemis configured to compare the resources consumed by the particular FSVMwith a capacity threshold for the particular FSVM.
 17. At least onenon-transitory computer readable medium encoded with instructions,which, when executed, cause a system to perform operations comprising:presenting a single namespace of storage items by a virtualizeddistributed file system, the virtualized distributed file systemincluding a cluster of host machines and at least one file servervirtual machine (FSVM) on each host machine in the cluster of hostmachines, the storage items stored in a storage pool distributed acrossthe cluster, the storage pool including an aggregation of local storagedevices of the host machines; processing, by the virtualized distributedfile system, an access operation directed to one of the storage items inthe single namespace, the one of the storage items stored in the storagepool; and scaling out the virtualized distributed file system responsiveto a load of the virtualized distributed file system, wherein thescaling out includes using a file server virtual machine manager to addan additional host machine to the cluster of host machines and using thefile server virtual machine manager to deploy an additional FSVM to theadditional host machine.
 18. The at least one non-transitory computerreadable medium of claim 17, wherein the operations further comprise:performing management operations at the FSVM manager for the virtualizeddistributed file system, the management operations including deployingthe at least one FSVM on each host machine, a number of the FSVMs in thecluster of host machines configured to form the single namespace of thestorage items.
 19. The at least one non-transitory computer readablemedium of claim 18, wherein the management operations further includeconfiguring each of the number of the FSVMs with a predetermined amountof memory and compute resources based on respective memory and computeresource parameters.
 20. The at least one non-transitory computerreadable medium of claim 17 wherein the operations further comprise:assigning one or more volume groups to each FSVM in the cluster of hostmachines, wherein each volume group includes a group of storage devicesof the storage pool; and creating one or more file shares including astorage capacity of the local storage devices of the storage pool,wherein each file share is stored on a respective volume group.
 21. Theat least one non-transitory computer readable medium of claim 20 whereinthe operations further comprise performing one or more input/output(I/O) transactions between each FSVM and the one or more file sharesstored on the assigned volume groups.
 22. The at least onenon-transitory computer readable medium of claim 20 wherein theoperations further comprise: reassigning one or more volume groups froma particular FSVM to another FSVM in response to the scale out of thevirtualized distributed file system to more evenly distribute the I/Otransactions among the FSVMs.
 23. The at least one non-transitorycomputer readable medium of claim 17, wherein the operations furthercomprise approximating the load of the virtualized distributed filesystem including determining resources consumed by a particular FSVM ofthe virtualized distributed file system.
 24. The at least onenon-transitory computer readable medium of claim 23, wherein theoperations further comprise comparing the resources consumed by theparticular FSVM with a capacity threshold for the particular FSVM.